gabrielct1/llama-aas-related-classification

Multilabel Classifier for Anabolic Steroid Comments (PT-BR)

LoRA adapter for multilabel classification of Brazilian Portuguese social media comments about anabolic steroid use.

• Adapter repository: gabrielct1/llama-aas-related-classification
• Base model: Meta Llama 3.1 8B (meta-llama/Meta-Llama-3.1-8B)

This model was developed as part of a research project at the Universidade Federal de Ouro Preto (UFOP), focused on understanding anabolic steroid discourse and self-reported experiences in Brazilian YouTube comments.

Overview

This model predicts two binary labels for each input comment:

• Autorrelato (Self-report)
• Sintomas / Efeitos Colaterais (Symptoms / Side Effects)

Training was performed with QLoRA on top of Meta Llama 3.1 8B (meta-llama/Meta-Llama-3.1-8B), using 4-bit NF4 quantization, multilabel stratified cross-validation, and per-label threshold tuning on validation F1.

Label Mapping

ID	Label	Description
0	Autorrelato (Self-report)	Includes comments that explicitly declare anabolic steroid use by the author themselves. This covers direct mentions of substances such as testosterone, oxandrolone, steroids, cycles, or brand names, as well as personal experience reports. It may also include references to dosage, duration of use, injection sites, or combinations with other substances, as long as the comment indicates self-use.
1	Sintomas / Efeitos Colaterais (Symptoms / Side Effects)	Includes comments that mention symptoms, adverse reactions, or side effects related to anabolic steroid use, regardless of who experienced them. This includes physical, emotional, or physiological changes such as acne, gynecomastia, hair loss, aggressiveness, infertility, among others. Comments that speculate about these effects should also be labeled in this category.

Comments may receive both labels simultaneously, a single label, or neither label.

Annotation Protocol and Agreement

• Annotation was performed independently by two authors.
• Inter-annotator agreement was measured with Cohen's Kappa.
• Overall agreement: k = 0.79 (substantial agreement).
• Self-report label: k = 0.91 (near-perfect agreement).
• Symptoms/Side Effects label: k = 0.77 (substantial agreement).
• Disagreements were resolved through consensus discussion.

Training Details

Item	Value
Method	QLoRA (PEFT)
Base model	Meta Llama 3.1 8B (meta-llama/Meta-Llama-3.1-8B)
Quantization	4-bit NF4 (BitsAndBytes)
Framework setup	QLoRA setup aligned with the Kelora/PEFT pipeline
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.1
LoRA target modules	q_proj, k_proj, v_proj, o_proj
Max sequence length	128
Learning rate	2e-4
Optimizer	AdamW
Weight decay	0.01
LR scheduler	Cosine schedule
Warmup ratio	0.05
Batch size	12
Max epochs	50 (early stopping enabled)
Validation protocol	5-fold multilabel stratified CV (cyclic: val=r, test=r+1)
Loss function	BCEWithLogitsLoss with class-specific pos_weight = (N - pos) / pos
Early stopping criterion	Validation loss (patience = 3, min delta = 1e-4)
Best checkpoint criterion	Macro F1
Post-processing	Per-label threshold optimization by validation F1 using precision-recall curves
Threshold search range	0.05 to 0.95

Quick Start

1) Installation

pip install torch transformers peft bitsandbytes accelerate huggingface_hub

2) Access requirements

This adapter depends on the gated base model Meta Llama 3.1 8B (meta-llama/Meta-Llama-3.1-8B).

Before running inference:

1. Request access to Meta Llama 3.1 8B (meta-llama/Meta-Llama-3.1-8B) on Hugging Face.
2. Accept Meta's license terms.
3. Login locally (huggingface-cli login) or set an environment token:

export HF_TOKEN="your_huggingface_token"

3) Inference from a Hugging Face Hub repo

import os
import json
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoModelForSequenceClassification, BitsAndBytesConfig
from peft import PeftModel

BASE_MODEL = "meta-llama/Meta-Llama-3.1-8B"
ADAPTER_REPO = "gabrielct1/llama-aas-related-classification"
HF_TOKEN = os.getenv("HF_TOKEN")

thr_path = hf_hub_download(repo_id=ADAPTER_REPO, filename="thresholds.json", token=HF_TOKEN)
with open(thr_path, "r", encoding="utf-8") as f:
  thresholds = torch.tensor(json.load(f)["thresholds"])

label_map = {
  0: "Autorrelato",
  1: "Sintomas / Efeitos Colaterais",
}

bnb_cfg = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_use_double_quant=True,
  bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO, token=HF_TOKEN)
if tokenizer.pad_token is None:
  tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForSequenceClassification.from_pretrained(
  BASE_MODEL,
  quantization_config=bnb_cfg,
  num_labels=2,
  device_map="auto",
  token=HF_TOKEN,
)
base_model.config.pad_token_id = tokenizer.pad_token_id

model = PeftModel.from_pretrained(base_model, ADAPTER_REPO, token=HF_TOKEN)
model.eval()

text = "Usei por algumas semanas e comecei a ter queda de cabelo e acne."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128).to(model.device)

with torch.no_grad():
  logits = model(**inputs).logits
  probs = torch.sigmoid(logits).squeeze(0).cpu()
  pred = (probs >= thresholds).int().tolist()

for i, name in label_map.items():
  print(f"{name}: prob={probs[i].item():.4f} pred={pred[i]}")

Note: this inference uses sigmoid plus per-label thresholds loaded from thresholds.json in the Hub repository.

If your environment does not support 4-bit loading, remove BitsAndBytesConfig(...) and load the base model in standard precision.

Data

The training process uses manually annotated comments in Brazilian Portuguese in the anabolic steroid domain.

Dataset overview:

• Unique comments: 478,397
• Unique users: 205,269
• Unique videos: 6,104
• Unique channels: 1,323
• Timeframe: 2020 to 2024

Limitations

• This model was trained on a specific domain (anabolic steroid comments) and may not generalize to other health domains.
• Very short, ironic, ambiguous, or context-dependent comments may reduce performance.
• The model may struggle to distinguish symptoms likely caused by anabolic steroid use from generic or unrelated symptoms when explicit causal context is missing.

Contact

• Name: Gabriel da Cunha Torres
• Institution: Universidade Federal de Ouro Preto (UFOP)
• Email: gabriel.ct@aluno.ufop.edu.br