Upload README.md with huggingface_hub

README.md

@@ -3,207 +3,411 @@ base_model: google/gemma-3-12b-it
 library_name: peft
 pipeline_tag: text-generation
 tags:
-- base_model:adapter:google/gemma-3-12b-it
-- lora
-- sft
-- transformers
-- trl
+  - base_model:adapter:google/gemma-3-12b-it
+  - lora
+  - sft
+  - transformers
+  - trl
 ---
 
-# Model Card for Model ID
+# Gemma3-MIAITS-Adapter
 
-<!-- Provide a quick summary of what the model is/does. -->
+**EN** | [LT](#lt-lietuvių)
 
+---
 
+## EN: English
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
+### Overview
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+**Gemma3-MIAITS-Adapter** is a LoRA adapter fine-tuned on top of [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) for Lithuanian-language misinformation classification, developed as part of the **MIAITS** project (_Melagingos informacijos automatinio identifikavimo tekstyno sukūrimas_ - Lithuanian Misinformation Automatic Identification Text Corpus).
 
-## Uses
+The model classifies Lithuanian news articles and statements into three categories:
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+| Label           | Meaning                           |
+| --------------- | --------------------------------- |
+| `Klaidinga`     | False / Fake information          |
+| `Manipuliatyvu` | Manipulative / Misleading content |
+| `Teisinga`      | True / Correct information        |
 
-### Direct Use
+---
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+### Architecture
 
-[More Information Needed]
+- **Base model**: [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) - Gemma 3 12B instruction-tuned
+- **Adapter type**: LoRA (PEFT) via QLoRA (4-bit NF4)
+- **Task**: Causal language modelling (text generation) - classification via generated JSON response
+- **LoRA rank (r)**: 32
+- **LoRA alpha**: 64
+- **LoRA dropout**: 0.1
+- **Target modules**: `q_proj`, `v_proj` (attention only)
+- **Bias**: none
+- **Quantization**: 4-bit NF4, compute dtype bfloat16, double quantization enabled
+- **PEFT version**: 0.18.1
 
-### Downstream Use [optional]
+---
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### Training Data
 
-[More Information Needed]
+**Source**: Lithuanian misinformation classification dataset.
 
-### Out-of-Scope Use
+**Labels** (3-class):
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+- `Klaidinga` - False
+- `Manipuliatyvu` - Manipulative
+- `Teisinga` - True
 
-[More Information Needed]
+**Text columns**: Each original row was expanded into 3 rows using `7-Statement`, `8-Statement_Context`, and `9-Full_text`. Validation and test sets use `9-Full_text` only.
 
-## Bias, Risks, and Limitations
+**Splits**: Stratified 80/10/10.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+| Split | Rows   |
+| ----- | ------ |
+| Train | 11,976 |
+| Val   | 499    |
+| Test  | 499    |
 
-[More Information Needed]
+---
 
-### Recommendations
+### Training Hyperparameters
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+| Parameter             | Value                       |
+| --------------------- | --------------------------- |
+| Learning rate         | 2e-5                        |
+| Scheduler             | Cosine (10% warmup)         |
+| Weight decay          | 0.05                        |
+| Epochs                | 3 (early stopping patience 2) |
+| Batch size            | 1                           |
+| Gradient accumulation | 32 (effective batch = 32)   |
+| Max sequence length   | 4,096                       |
+| Precision             | BF16                        |
+| Max new tokens (eval) | 512                         |
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+**Selected checkpoint**: epoch 1 (best eval_loss). Runtime: ~11.6h.
 
-## How to Get Started with the Model
+---
 
-Use the code below to get started with the model.
+### Prompt Format
 
-[More Information Needed]
+The system prompt instructs the model (in Lithuanian) to classify the text and respond in JSON:
 
-## Training Details
+```json
+{"label": "<label>", "justification": "<1-sentence explanation>"}
+```
 
-### Training Data
+Ground truth justifications from the `17-Justification` column of the source data were used as assistant responses during training.
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+---
 
-[More Information Needed]
+### Performance
 
-### Training Procedure
+Evaluated on the MIAITS test set (499 rows, 3 classes). Best checkpoint: epoch 1 (by eval_loss).
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+| Metric             | Val    | Test   |
+| ------------------ | ------ | ------ |
+| Accuracy           | 67.1%  | 68.0%  |
+| Macro F1           | 0.647  | 0.662  |
+| JSON parse failures | 2.4%  | 1.8%   |
 
-#### Preprocessing [optional]
+**Epoch progression**:
 
-[More Information Needed]
+| Epoch    | Val Acc | Val Macro F1 | Test Acc | Test Macro F1 |
+| -------- | ------- | ------------ | -------- | ------------- |
+| 0 (base) | 42.3%   | 0.384        | 41.6%    | 0.399         |
+| 1        | 65.3%   | 0.610        | 66.0%    | 0.641         |
+| 2        | 66.7%   | 0.643        | 68.2%    | 0.668         |
+| 3        | 66.9%   | 0.642        | 68.2%    | 0.667         |
 
+**Per-class metrics (Test)**:
 
-#### Training Hyperparameters
+| Class         | Precision | Recall | F1    |
+| ------------- | --------- | ------ | ----- |
+| Klaidinga     | 0.637     | 0.717  | 0.675 |
+| Manipuliatyvu | 0.514     | 0.400  | 0.450 |
+| Teisinga      | 0.843     | 0.881  | 0.862 |
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+> **Note:** `Manipuliatyvu` is the hardest class (F1 0.450). `Teisinga` is the easiest (F1 0.862). ~2% of outputs failed JSON parsing.
 
-#### Speeds, Sizes, Times [optional]
+---
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+### Intended Use
 
-[More Information Needed]
+- Secondary signal in an ensemble alongside ModernBERT
+- Text normalization pre-processing (restoring perturbed Lithuanian text before classification)
+- Research on generative LLMs for Baltic/Eastern European misinformation detection
 
-## Evaluation
+### Limitations
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+- Trained exclusively on Lithuanian-language data; not suitable for other languages
+- `Manipuliatyvu` class has notably low recall (0.400)
+- Produces occasional JSON parse failures (~2%)
 
-### Testing Data, Factors & Metrics
+---
 
-#### Testing Data
+### Usage
+
+**Hardware requirements:**
+
+- bfloat16 (default): ~24 GB VRAM
+- 4-bit quantization (`--load-in-4bit`): ~8-10 GB VRAM (recommended for consumer GPUs)
+
+```python
+import torch
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+
+BASE_MODEL   = "google/gemma-3-12b-it"
+ADAPTER_PATH = "VSSA-SDSA/LT_AI_FakeNews_LLM"
+
+# Optional: 4-bit quantization to reduce VRAM usage
+quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+
+base_model = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL,
+    quantization_config=quantization_config,  # remove for full bfloat16
+    device_map="auto",
+)
+
+model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
+model.eval()
+
+
+def classify(text: str, max_new_tokens: int = 512) -> dict:
+    messages = [
+        {
+            "role": "system",
+            "content": (
+                "Esi lietuviškos žiniasklaidos analizės įrankis. "
+                "Klasifikuok pateiktą tekstą į vieną iš trijų kategorijų: "
+                "Klaidinga, Manipuliatyvu, Teisinga. "
+                'Atsakyk JSON formatu: {"label": "<kategorija>", "justification": "<vienas sakinys>"}'
+            ),
+        },
+        {"role": "user", "content": text},
+    ]
+    inputs = tokenizer.apply_chat_template(
+        messages, return_tensors="pt", add_generation_prompt=True
+    ).to(model.device)
+    with torch.inference_mode():
+        output_ids = model.generate(
+            inputs,
+            max_new_tokens=max_new_tokens,
+            do_sample=False,
+            repetition_penalty=1.1,
+        )
+    new_tokens = output_ids[0][inputs.shape[1]:]
+    return tokenizer.decode(new_tokens, skip_special_tokens=True)
+
+
+text = "Mokslininkai įrodė, kad žemė yra plokščia ir NASA slepia tiesą."
+print(classify(text))
+```
 
-<!-- This should link to a Dataset Card if possible. -->
+---
 
-[More Information Needed]
+## LT: Lietuvių
 
-#### Factors
+### Apžvalga
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+**Gemma3-MIAITS-Adapter** - tai LoRA adapteris, suderintas ant [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) pagrindu lietuviškos dezinformacijos klasifikavimui, sukurtas **MIAITS** projekto (_Melagingos informacijos automatinio identifikavimo tekstyno sukūrimas_) rėmuose.
 
-[More Information Needed]
+Modelis klasifikuoja lietuviškus naujienų straipsnius ir teiginius į tris kategorijas:
 
-#### Metrics
+| Žyma            | Reikšmė                                 |
+| --------------- | --------------------------------------- |
+| `Klaidinga`     | Klaidinga / melaginga informacija       |
+| `Manipuliatyvu` | Manipuliatyvi / klaidinanti informacija |
+| `Teisinga`      | Teisinga informacija                    |
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+---
 
-[More Information Needed]
+### Architektūra
 
-### Results
+- **Bazinis modelis**: [`google/gemma-3-12b-it`](https://huggingface.co/google/gemma-3-12b-it) - Gemma 3 12B instrukcinė versija
+- **Adapterio tipas**: LoRA (PEFT) per QLoRA (4 bitų NF4)
+- **Užduotis**: Priežastinis kalbos modeliavimas (teksto generavimas) - klasifikavimas per sugeneruotą JSON atsakymą
+- **LoRA rangas (r)**: 32
+- **LoRA alpha**: 64
+- **LoRA dropout**: 0,1
+- **Tiksliniai moduliai**: `q_proj`, `v_proj` (tik dėmesio sluoksniai)
+- **Bias**: nėra
+- **Kvantizacija**: 4 bitų NF4, skaičiavimo tipas bfloat16, dviguba kvantizacija įjungta
+- **PEFT versija**: 0.18.1
 
-[More Information Needed]
+---
 
-#### Summary
+### Mokymo duomenys
 
+**Šaltinis**: Lietuviškos dezinformacijos klasifikavimo duomenų rinkinys.
 
+**Žymos** (3 klasės):
 
-## Model Examination [optional]
+- `Klaidinga` - Klaidinga
+- `Manipuliatyvu` - Manipuliatyvu
+- `Teisinga` - Teisinga
 
-<!-- Relevant interpretability work for the model goes here -->
+**Teksto stulpeliai**: Kiekviena originali eilutė buvo išplėsta į 3 eilutes naudojant `7-Statement`, `8-Statement_Context` ir `9-Full_text`. Validavimo ir testavimo rinkiniai naudoja tik `9-Full_text`.
 
-[More Information Needed]
+**Padalijimas**: Stratifikuotas 80/10/10.
 
-## Environmental Impact
+| Rinkinys    | Eilutės |
+| ----------- | ------- |
+| Mokymas     | 11 976  |
+| Validavimas | 499     |
+| Testavimas  | 499     |
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+---
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+### Mokymo hiperparametrai
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+| Parametras            | Reikšmė                           |
+| --------------------- | --------------------------------- |
+| Mokymosi greitis      | 2e-5                              |
+| Planuoklis            | Kosinusinis (10 % apšilimas)      |
+| Svorių atitolimas     | 0,05                              |
+| Epochos               | 3 (ankstyvas stabdymas, kantryb. 2) |
+| Paketų dydis          | 1                                 |
+| Gradiento kaupimas    | 32 (efektyvus paketas = 32)       |
+| Maks. sekos ilgis     | 4 096                             |
+| Tikslumas             | BF16                              |
+| Maks. naujų žetonų (eval) | 512                           |
 
-## Technical Specifications [optional]
+**Pasirinktas kontrolinis taškas**: 1 epocha (geriausias eval_loss). Trukmė: ~11,6 val.
 
-### Model Architecture and Objective
+---
 
-[More Information Needed]
+### Užklausos formatas
 
-### Compute Infrastructure
+Sistemos užklausa nurodo modeliui (lietuvių kalba) klasifikuoti tekstą ir atsakyti JSON formatu:
 
-[More Information Needed]
+```json
+{"label": "<žyma>", "justification": "<vienas sakinys>"}
+```
 
-#### Hardware
+Mokymo metu kaip asistentu atsakymai naudoti tikrojo šaltinio pagrindinimai iš `17-Justification` stulpelio.
 
-[More Information Needed]
+---
 
-#### Software
+### Rezultatai
 
-[More Information Needed]
+Įvertinta su MIAITS testavimo rinkiniu (499 eilutės, 3 klasės). Geriausias kontrolinis taškas: 1 epocha (pagal eval_loss).
 
-## Citation [optional]
+| Metrika                | Validavimas | Testavimas |
+| ---------------------- | ----------- | ---------- |
+| Tikslumas              | 67,1 %      | 68,0 %     |
+| Makro F1               | 0,647       | 0,662      |
+| JSON apdorojimo klaidos | 2,4 %      | 1,8 %      |
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+**Epochų progresija**:
 
-**BibTeX:**
+| Epocha      | Val. tiksl. | Val. makro F1 | Test. tiksl. | Test. makro F1 |
+| ----------- | ----------- | ------------- | ------------ | -------------- |
+| 0 (bazinis) | 42,3 %      | 0,384         | 41,6 %       | 0,399          |
+| 1           | 65,3 %      | 0,610         | 66,0 %       | 0,641          |
+| 2           | 66,7 %      | 0,643         | 68,2 %       | 0,668          |
+| 3           | 66,9 %      | 0,642         | 68,2 %       | 0,667          |
 
-[More Information Needed]
+**Tikslumas pagal klasę (testavimas)**:
 
-**APA:**
+| Klasė         | Tikslumas | Atkūrimas | F1    |
+| ------------- | --------- | --------- | ----- |
+| Klaidinga     | 0,637     | 0,717     | 0,675 |
+| Manipuliatyvu | 0,514     | 0,400     | 0,450 |
+| Teisinga      | 0,843     | 0,881     | 0,862 |
 
-[More Information Needed]
+> **Pastaba:** `Manipuliatyvu` yra sunkiausia klasė (F1 0,450). `Teisinga` lengviausia (F1 0,862). ~2 % išvesties nepavyko apdoroti JSON formatu.
 
-## Glossary [optional]
+---
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+### Numatytas naudojimas
 
-[More Information Needed]
+- Papildomas signalas ansamblyje kartu su ModernBERT
+- Teksto normalizavimo išankstinis apdorojimas (sugadintų lietuviškų tekstų atstatymas prieš klasifikavimą)
+- Generatyviųjų kalbos modelių dezinformacijos aptikimo tyrimai Baltijos ir Rytų Europos regione
 
-## More Information [optional]
+### Apribojimai
 
-[More Information Needed]
+- Modelis apmokytas išimtinai lietuviškais duomenimis; kitoms kalboms netinka
+- `Manipuliatyvu` klasės atkūrimas ypač žemas (0,400)
+- Kartais nepavyksta apdoroti JSON išvesties (~2 %)
 
-## Model Card Authors [optional]
+---
 
-[More Information Needed]
+### Naudojimas
+
+**Aparatūros reikalavimai:**
+
+- bfloat16 (numatytasis): ~24 GB VRAM
+- 4 bitų kvantizacija: ~8-10 GB VRAM (rekomenduojama vartotojų GPU)
+
+```python
+import torch
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+
+BASE_MODEL   = "google/gemma-3-12b-it"
+ADAPTER_PATH = "VSSA-SDSA/LT_AI_FakeNews_LLM"
+
+# Neprivaloma: 4 bitų kvantizacija, siekiant sumažinti VRAM naudojimą
+quantization_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_compute_dtype=torch.bfloat16,
+    bnb_4bit_use_double_quant=True,
+    bnb_4bit_quant_type="nf4",
+)
+
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+
+base_model = AutoModelForCausalLM.from_pretrained(
+    BASE_MODEL,
+    quantization_config=quantization_config,  # pašalinti norint naudoti visą bfloat16
+    device_map="auto",
+)
+
+model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
+model.eval()
+
+
+def classify(text: str, max_new_tokens: int = 512) -> str:
+    messages = [
+        {
+            "role": "system",
+            "content": (
+                "Esi lietuviškos žiniasklaidos analizės įrankis. "
+                "Klasifikuok pateiktą tekstą į vieną iš trijų kategorijų: "
+                "Klaidinga, Manipuliatyvu, Teisinga. "
+                'Atsakyk JSON formatu: {"label": "<kategorija>", "justification": "<vienas sakinys>"}'
+            ),
+        },
+        {"role": "user", "content": text},
+    ]
+    inputs = tokenizer.apply_chat_template(
+        messages, return_tensors="pt", add_generation_prompt=True
+    ).to(model.device)
+    with torch.inference_mode():
+        output_ids = model.generate(
+            inputs,
+            max_new_tokens=max_new_tokens,
+            do_sample=False,
+            repetition_penalty=1.1,
+        )
+    new_tokens = output_ids[0][inputs.shape[1]:]
+    return tokenizer.decode(new_tokens, skip_special_tokens=True)
+
+
+text = "Mokslininkai įrodė, kad žemė yra plokščia ir NASA slepia tiesą."
+print(classify(text))
+```
 
-## Model Card Contact
+---
 
-[More Information Needed]
 ### Framework versions
 
-- PEFT 0.18.1
+- PEFT 0.18.1