# Models

Detail for the 21 checkpoints. The system description and results are in the
[model card](README.md), the combiner in [`stacking_ensemble/`](stacking_ensemble/README.md), and the
generated data in [`data/`](data/README.md).

Encoders share one recipe, recorded in full in each model's `best.json`: max_len 128, batch 16,
lr 2e-5, weight decay 0.01, seed 42, 12 epochs with patience 3, cross-entropy unless noted. Label
order is `["Against","Favor","None"]`, so `id2label` is 0=Against, 1=Favor, 2=None. `prep_mode:
preserve` keeps hashtags and elongation, `strip` removes them. Track-2 models prepend a short target
description to help cross-target transfer.

LoRA adapters share r=16, α=32 over all 7 projection matrices (q, k, v, o, gate, up, down), with
inference scoring the log-probability of the label continuation.

`Dev F` is the best dev *F*<sub>avg2</sub> reached during training. For Track 1 that is in-domain; for
Track 2 it is the held-out Women Empowerment target. Neither is comparable to the test scores in the
model card.

## Encoders

Load with `AutoModelForSequenceClassification.from_pretrained(repo, subfolder="models/encoders/<name>")`.

| Subfolder | Base | Dev *F* | Ep. | Role |
|---|---|---:|---:|---|
| `t1_marbert` | UBC-NLP/MARBERTv2 | 0.8360 | 3 | main Track-1 encoder, also the retrieval embedder |
| `t1_arabert` | aubmindlab/bert-base-arabertv02-twitter | 0.8351 | 2 | ensemble member |
| `t1_arabert_strip` | aubmindlab/bert-base-arabertv02-twitter | 0.8385 | 7 | preprocessing variant (`strip`) |
| `t1_marbert_strip` | UBC-NLP/MARBERTv2 | 0.8311 | 6 | preprocessing variant (`strip`) |
| `t1_camelbert` | CAMeL-Lab/bert-base-arabic-camelbert-mix | 0.7904 | 3 | alternative family |
| `t1_aramodern` | NAMAA-Space/AraModernBert-Base-V1.0 | 0.7897 | 10 | alternative family |
| `t2_marbert` | UBC-NLP/MARBERTv2 | 0.8195 | 2 | main Track-2 encoder and retrieval embedder |
| `t2_marbert_strip` | UBC-NLP/MARBERTv2 | 0.8147 | 3 | preprocessing variant (`strip`) |
| `t2_marbert_weighted` | UBC-NLP/MARBERTv2 | 0.8088 | 3 | class-weighted loss, rejected |
| `t2_marbert_aug` | UBC-NLP/MARBERTv2 | 0.7930 | 3 | trained on `data/track2/train_aug.csv` |
| `t2_arabert` | aubmindlab/bert-base-arabertv02-twitter | 0.7837 | 3 | ensemble member |
| `t2_marbert_focal` | UBC-NLP/MARBERTv2 | 0.7794 | 3 | focal loss (γ=2), rejected |
| `t2_camelbert` | CAMeL-Lab/bert-base-arabic-camelbert-mix | 0.7506 | 2 | alternative family |
| `t2_aramodern` | NAMAA-Space/AraModernBert-Base-V1.0 | 0.7308 | 5 | alternative family |

MARBERTv2's lead over AraBERT-twitter is 0.1 points on Track 1 and 3.6 on Track 2. Both loss changes
did worse than plain cross-entropy. This is experiment 1 in the model card.

## LoRA adapters

Attach with `PeftModel.from_pretrained(base, repo, subfolder="models/lora/<name>")`. The base comes
from its own repository.

| Subfolder | Base | Trained on | Role |
|---|---|---|---|
| `allam_t2` | ALLaM-AI/ALLaM-7B-Instruct-preview | Track-2 train (Covid, Digital) | main Track-2 generative member, WE-dev *F*<sub>avg2</sub> 0.883, blended 0.5 with the few-shot member |
| `allam_full` | ALLaM-7B | Track-1 train | counterweight member; its Favor lean offsets the few-shot model's Against lean |
| `allam_v2` | ALLaM-7B | Track-1 train | second counterweight member |
| `allam_loo` | ALLaM-7B | Track-1 train, leave-one-target-out | transfer check |
| `allam_real` | ALLaM-7B | `data/external/pool_real_all.csv` | real-pool arm of the counterweight comparison |
| `allam_style` | ALLaM-7B | `data/synth/t1_train_style.csv` | style-pool arm; retraining for accuracy cost 0.0027 (experiment 7) |
| `qwen_t2` | Qwen/Qwen2.5-7B-Instruct | Track-2 train | decorrelation member, rejected at −0.0373 (experiment 7) |

## Notes

* Base models are referenced, not re-hosted. ALLaM-7B, Qwen2.5-7B and the served models (Gemma-4-31B,
  Qwen3.6-35B) are third party; only our adapters and fine-tuned encoders are here.
* Four encoders used as probability sources were never saved: AraELECTRA, XLM-R-large, ARBERTv2 and
  AraBERT-large. Their outputs are in `stacking_ensemble/probs/`, and `code/` rebuilds them by
  overriding the base id (see [`code/README.md`](code/README.md)).
* The Qwen3.6-35B NVFP4 checkpoint needed an `lm_head` dequantization patch to serve. Gemma-4-31B
  NVFP4 was the main few-shot backend.