README.md

---
library_name: transformers
pipeline_tag: text-classification
base_model: FacebookAI/xlm-roberta-base
tags:
- text-classification
- binary-classification
- amis
- agriculture
language: multilingual
---

# AMIS Commodity Classifier

This model repository contains artifacts from an AMIS commodity relevance classifier training run.
It includes the Transformer model, any configured TF-IDF or sentence-embedding baselines, prediction files, and the training report.

- Dataset: `faodl/amis-agri-vegetable_oils`
- Dataset subset: ``
- Dataset revision: `main`
- Text column: `chunk_text`
- Label column: `label`
- Transformer: `FacebookAI/xlm-roberta-base`
- Generated at: `2026-06-09T16:39:33.521615+00:00`

## Dataset Summary

| Split | Rows | Label 0 | Label 1 | Unique groups | Mean text length |
| --- | ---: | ---: | ---: | ---: | ---: |
| train | 5382 | 4040 | 1342 | 2395 | 706.0 |
| validation | 1186 | 900 | 286 | 513 | 711.4 |
| test | 1082 | 847 | 235 | 514 | 699.9 |

## Threshold Comparison on Validation Split

Validation metrics document threshold selection and tuning behavior; test metrics remain the primary estimate of out-of-sample performance.

| Model | Threshold | Accuracy | Precision | Recall | F1 | ROC AUC | Average precision |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| logistic_tfidf | 0.500 | 0.879 | 0.738 | 0.769 | 0.753 | 0.929 | 0.804 |
| logistic_tfidf | 0.486 | 0.880 | 0.732 | 0.794 | 0.762 | 0.929 | 0.804 |
| xgboost_tfidf | 0.500 | 0.865 | 0.797 | 0.591 | 0.679 | 0.926 | 0.816 |
| xgboost_tfidf | 0.357 | 0.882 | 0.763 | 0.741 | 0.752 | 0.926 | 0.816 |
| embedding-logistic_sentence_embeddings | 0.500 | 0.847 | 0.632 | 0.881 | 0.736 | 0.925 | 0.802 |
| embedding-logistic_sentence_embeddings | 0.620 | 0.871 | 0.711 | 0.783 | 0.745 | 0.925 | 0.802 |
| embedding-svm_sentence_embeddings | 0.500 | 0.853 | 0.767 | 0.563 | 0.649 | 0.918 | 0.787 |
| embedding-svm_sentence_embeddings | 0.357 | 0.865 | 0.685 | 0.815 | 0.744 | 0.918 | 0.787 |
| embedding-lightgbm_sentence_embeddings | 0.500 | 0.881 | 0.760 | 0.741 | 0.750 | 0.930 | 0.829 |
| embedding-lightgbm_sentence_embeddings | 0.318 | 0.879 | 0.723 | 0.804 | 0.762 | 0.930 | 0.829 |
| transformer | 0.500 | 0.896 | 0.762 | 0.829 | 0.794 | 0.940 | 0.872 |
| transformer | 0.625 | 0.910 | 0.816 | 0.808 | 0.812 | 0.940 | 0.872 |

## Threshold Comparison on Test Split

| Model | Threshold | Accuracy | Precision | Recall | F1 | ROC AUC | Average precision |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| logistic_tfidf | 0.500 | 0.896 | 0.766 | 0.753 | 0.760 | 0.937 | 0.817 |
| logistic_tfidf | 0.486 | 0.896 | 0.756 | 0.766 | 0.761 | 0.937 | 0.817 |
| xgboost_tfidf | 0.500 | 0.878 | 0.772 | 0.621 | 0.689 | 0.923 | 0.785 |
| xgboost_tfidf | 0.357 | 0.882 | 0.738 | 0.706 | 0.722 | 0.923 | 0.785 |
| embedding-logistic_sentence_embeddings | 0.500 | 0.835 | 0.583 | 0.834 | 0.687 | 0.919 | 0.763 |
| embedding-logistic_sentence_embeddings | 0.620 | 0.857 | 0.648 | 0.745 | 0.693 | 0.919 | 0.763 |
| embedding-svm_sentence_embeddings | 0.500 | 0.871 | 0.768 | 0.579 | 0.660 | 0.908 | 0.758 |
| embedding-svm_sentence_embeddings | 0.357 | 0.841 | 0.611 | 0.740 | 0.669 | 0.908 | 0.758 |
| embedding-lightgbm_sentence_embeddings | 0.500 | 0.879 | 0.745 | 0.672 | 0.707 | 0.921 | 0.793 |
| embedding-lightgbm_sentence_embeddings | 0.318 | 0.872 | 0.689 | 0.753 | 0.720 | 0.921 | 0.793 |
| transformer | 0.500 | 0.891 | 0.741 | 0.766 | 0.753 | 0.930 | 0.820 |
| transformer | 0.625 | 0.894 | 0.759 | 0.749 | 0.754 | 0.930 | 0.820 |

## Confusion Matrices on Test Split

Rows are true labels and columns are predicted labels.

### logistic_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 793 | 54 |
| RELEVANT | 58 | 177 |

### logistic_tfidf at threshold 0.486

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 789 | 58 |
| RELEVANT | 55 | 180 |

### xgboost_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 804 | 43 |
| RELEVANT | 89 | 146 |

### xgboost_tfidf at threshold 0.357

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 788 | 59 |
| RELEVANT | 69 | 166 |

### embedding-logistic_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 707 | 140 |
| RELEVANT | 39 | 196 |

### embedding-logistic_sentence_embeddings at threshold 0.620

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 752 | 95 |
| RELEVANT | 60 | 175 |

### embedding-svm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 806 | 41 |
| RELEVANT | 99 | 136 |

### embedding-svm_sentence_embeddings at threshold 0.357

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 736 | 111 |
| RELEVANT | 61 | 174 |

### embedding-lightgbm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 793 | 54 |
| RELEVANT | 77 | 158 |

### embedding-lightgbm_sentence_embeddings at threshold 0.318

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 767 | 80 |
| RELEVANT | 58 | 177 |

### transformer at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 784 | 63 |
| RELEVANT | 55 | 180 |

### transformer at threshold 0.625

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 791 | 56 |
| RELEVANT | 59 | 176 |


## Validation-Tuned Thresholds

- `logistic_tfidf`: threshold `0.486` (validation F1 `0.762`); test F1 change vs 0.5: `+0.001`.
- `xgboost_tfidf`: threshold `0.357` (validation F1 `0.752`); test F1 change vs 0.5: `+0.033`.
- `embedding-logistic_sentence_embeddings`: threshold `0.620` (validation F1 `0.745`); test F1 change vs 0.5: `+0.007`.
- `embedding-svm_sentence_embeddings`: threshold `0.357` (validation F1 `0.744`); test F1 change vs 0.5: `+0.009`.
- `embedding-lightgbm_sentence_embeddings`: threshold `0.318` (validation F1 `0.762`); test F1 change vs 0.5: `+0.013`.
- `transformer`: threshold `0.625` (validation F1 `0.812`); test F1 change vs 0.5: `+0.001`.

## Artifacts

- `logistic_tfidf`: `/content/agri-vegetable_oils-classifier/baselines/logistic`
- `xgboost_tfidf`: `/content/agri-vegetable_oils-classifier/baselines/xgboost`
- `embedding-logistic_sentence_embeddings`: `/content/agri-vegetable_oils-classifier/baselines/embedding-logistic`
- `embedding-svm_sentence_embeddings`: `/content/agri-vegetable_oils-classifier/baselines/embedding-svm`
- `embedding-lightgbm_sentence_embeddings`: `/content/agri-vegetable_oils-classifier/baselines/embedding-lightgbm`
- `transformer`: `/content/agri-vegetable_oils-classifier/transformer`

## Inference

Install the runtime dependencies:

```bash
pip install transformers torch huggingface_hub pandas joblib scikit-learn xgboost lightgbm
```

### Transformer

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "faodl/agri-vegetable_oils-classifier"

texts = [
    "Rice export prices increased after new procurement rules were announced.",
    "The finance ministry released its monthly fuel tax bulletin.",
]

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, subfolder="transformer")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, subfolder="transformer")
threshold = float(getattr(model.config, "threshold", 0.5))

encoded = tokenizer(
    texts,
    truncation=True,
    padding=True,
    max_length=256,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**encoded).logits
    probabilities = torch.softmax(logits, dim=-1)[:, 1].tolist()

for text, probability in zip(texts, probabilities):
    label = model.config.id2label[int(probability >= threshold)]
    print({"text": text, "probability_positive": probability, "label": label})
```

### TF-IDF Baselines

Available baseline names in this run: "logistic", "xgboost".

```python
import json
import joblib
from huggingface_hub import hf_hub_download

MODEL_ID = "faodl/agri-vegetable_oils-classifier"
BASELINE = "logistic"

texts = [
    "Maize production forecasts were revised after delayed rains.",
    "The central bank published new exchange rate statistics.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}_tfidf.joblib",
)
report_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename="report.json",
)

pipeline = joblib.load(model_path)
with open(report_path, encoding="utf-8") as handle:
    report = json.load(handle)

threshold = next(
    result["validation_best_threshold"]["threshold"]
    for result in report["results"]
    if result["model_type"] == f"{BASELINE}_tfidf"
)

probabilities = pipeline.predict_proba(texts)[:, 1]
for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})
```

### Sentence-Embedding Baselines

Available embedding baseline names in this run: "embedding-logistic", "embedding-svm", "embedding-lightgbm".

```python
import joblib
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoModel, AutoTokenizer

MODEL_ID = "faodl/agri-vegetable_oils-classifier"
BASELINE = "embedding-logistic"

texts = [
    "Wheat export inspections rose as demand from importers increased.",
    "The sports ministry announced a new stadium renovation plan.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}.joblib",
)
artifact = joblib.load(model_path)
tokenizer = AutoTokenizer.from_pretrained(artifact["embedding_model_name"])
encoder = AutoModel.from_pretrained(artifact["embedding_model_name"])
encoder.eval()

encoded_batches = []
batch_size = artifact.get("embedding_batch_size", 64)
for start in range(0, len(texts), batch_size):
    batch_texts = texts[start : start + batch_size]
    inputs = tokenizer(
        batch_texts,
        padding=True,
        truncation=True,
        max_length=artifact.get("embedding_max_length", 256),
        return_tensors="pt",
    )
    with torch.no_grad():
        outputs = encoder(**inputs)
    token_embeddings = outputs.last_hidden_state
    attention_mask = inputs["attention_mask"].unsqueeze(-1).to(token_embeddings.dtype)
    embeddings = (token_embeddings * attention_mask).sum(dim=1)
    embeddings = embeddings / attention_mask.sum(dim=1).clamp(min=1e-9)
    if artifact.get("normalize_embeddings", True):
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    encoded_batches.append(embeddings)
embeddings = torch.cat(encoded_batches).numpy()
probabilities = artifact["classifier"].predict_proba(embeddings)[:, 1]
threshold = artifact["validation_best_threshold"]["threshold"]

for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})
```

## Files

- `REPORT.md`: Markdown report for this training run.
- `report.json`: Machine-readable report containing metrics and thresholds.
- `transformer/`: Fine-tuned Transformer artifacts, when Transformer training is enabled.
- `baselines/`: TF-IDF and sentence-embedding baseline artifacts, when baseline training is enabled.
- `*/validation_predictions.csv` and `*/test_predictions.csv`: Split-level predictions.