README.md

---
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert/distilbert-base-multilingual-cased
tags:
- text-classification
- binary-classification
- amis
- agriculture
language: multilingual
---

# AMIS Commodity Classifier

This model repository contains artifacts from an AMIS commodity relevance classifier training run.
It includes the Transformer model, any configured TF-IDF or sentence-embedding baselines, prediction files, and the training report.

- Dataset: `faodl/amis-agri-soybeans`
- Dataset subset: ``
- Text column: `chunk_text`
- Label column: `label`
- Transformer: `distilbert/distilbert-base-multilingual-cased`
- Generated at: `2026-05-19T20:13:44.207534+00:00`

## Dataset Summary

| Split | Rows | Label 0 | Label 1 | Unique groups | Mean text length |
| --- | ---: | ---: | ---: | ---: | ---: |
| train | 4745 | 3860 | 885 | 2244 | 702.4 |
| validation | 1034 | 782 | 252 | 481 | 710.3 |
| test | 1074 | 889 | 185 | 482 | 708.6 |

## Threshold Comparison on Test Split

| Model | Threshold | Accuracy | Precision | Recall | F1 | ROC AUC | Average precision |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| logistic_tfidf | 0.500 | 0.944 | 0.805 | 0.892 | 0.846 | 0.967 | 0.914 |
| logistic_tfidf | 0.454 | 0.941 | 0.785 | 0.908 | 0.842 | 0.967 | 0.914 |
| xgboost_tfidf | 0.500 | 0.954 | 0.895 | 0.832 | 0.863 | 0.964 | 0.896 |
| xgboost_tfidf | 0.549 | 0.955 | 0.905 | 0.827 | 0.864 | 0.964 | 0.896 |
| embedding-logistic_sentence_embeddings | 0.500 | 0.939 | 0.753 | 0.957 | 0.843 | 0.988 | 0.951 |
| embedding-logistic_sentence_embeddings | 0.647 | 0.954 | 0.837 | 0.914 | 0.873 | 0.988 | 0.951 |
| embedding-svm_sentence_embeddings | 0.500 | 0.957 | 0.884 | 0.865 | 0.874 | 0.988 | 0.949 |
| embedding-svm_sentence_embeddings | 0.379 | 0.955 | 0.848 | 0.903 | 0.874 | 0.988 | 0.949 |
| embedding-lightgbm_sentence_embeddings | 0.500 | 0.959 | 0.894 | 0.865 | 0.879 | 0.985 | 0.950 |
| embedding-lightgbm_sentence_embeddings | 0.429 | 0.959 | 0.890 | 0.870 | 0.880 | 0.985 | 0.950 |
| transformer | 0.500 | 0.954 | 0.882 | 0.849 | 0.865 | 0.976 | 0.929 |
| transformer | 0.493 | 0.955 | 0.883 | 0.854 | 0.868 | 0.976 | 0.929 |

## Confusion Matrices on Test Split

Rows are true labels and columns are predicted labels.

### logistic_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 849 | 40 |
| RELEVANT | 20 | 165 |

### logistic_tfidf at threshold 0.454

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 843 | 46 |
| RELEVANT | 17 | 168 |

### xgboost_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 871 | 18 |
| RELEVANT | 31 | 154 |

### xgboost_tfidf at threshold 0.549

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 873 | 16 |
| RELEVANT | 32 | 153 |

### embedding-logistic_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 831 | 58 |
| RELEVANT | 8 | 177 |

### embedding-logistic_sentence_embeddings at threshold 0.647

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 856 | 33 |
| RELEVANT | 16 | 169 |

### embedding-svm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 868 | 21 |
| RELEVANT | 25 | 160 |

### embedding-svm_sentence_embeddings at threshold 0.379

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 859 | 30 |
| RELEVANT | 18 | 167 |

### embedding-lightgbm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 870 | 19 |
| RELEVANT | 25 | 160 |

### embedding-lightgbm_sentence_embeddings at threshold 0.429

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 869 | 20 |
| RELEVANT | 24 | 161 |

### transformer at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 868 | 21 |
| RELEVANT | 28 | 157 |

### transformer at threshold 0.493

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 868 | 21 |
| RELEVANT | 27 | 158 |


## Validation-Tuned Thresholds

- `logistic_tfidf`: threshold `0.454` (validation F1 `0.870`); test F1 change vs 0.5: `-0.004`.
- `xgboost_tfidf`: threshold `0.549` (validation F1 `0.900`); test F1 change vs 0.5: `+0.002`.
- `embedding-logistic_sentence_embeddings`: threshold `0.647` (validation F1 `0.851`); test F1 change vs 0.5: `+0.031`.
- `embedding-svm_sentence_embeddings`: threshold `0.379` (validation F1 `0.840`); test F1 change vs 0.5: `+0.000`.
- `embedding-lightgbm_sentence_embeddings`: threshold `0.429` (validation F1 `0.847`); test F1 change vs 0.5: `+0.001`.
- `transformer`: threshold `0.493` (validation F1 `0.924`); test F1 change vs 0.5: `+0.003`.

## Artifacts

- `logistic_tfidf`: `/content/agri-soybeans-classifier/baselines/logistic`
- `xgboost_tfidf`: `/content/agri-soybeans-classifier/baselines/xgboost`
- `embedding-logistic_sentence_embeddings`: `/content/agri-soybeans-classifier/baselines/embedding-logistic`
- `embedding-svm_sentence_embeddings`: `/content/agri-soybeans-classifier/baselines/embedding-svm`
- `embedding-lightgbm_sentence_embeddings`: `/content/agri-soybeans-classifier/baselines/embedding-lightgbm`
- `transformer`: `/content/agri-soybeans-classifier/transformer`

## Inference

Install the runtime dependencies:

```bash
pip install transformers torch huggingface_hub pandas joblib scikit-learn xgboost sentence-transformers lightgbm
```

### Transformer

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "faodl/agri-soybeans-classifier"

texts = [
    "Rice export prices increased after new procurement rules were announced.",
    "The finance ministry released its monthly fuel tax bulletin.",
]

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, subfolder="transformer")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, subfolder="transformer")
threshold = float(getattr(model.config, "threshold", 0.5))

encoded = tokenizer(
    texts,
    truncation=True,
    padding=True,
    max_length=256,
    return_tensors="pt",
)

with torch.no_grad():
    logits = model(**encoded).logits
    probabilities = torch.softmax(logits, dim=-1)[:, 1].tolist()

for text, probability in zip(texts, probabilities):
    label = model.config.id2label[int(probability >= threshold)]
    print({"text": text, "probability_positive": probability, "label": label})
```

### TF-IDF Baselines

Available baseline names in this run: "logistic", "xgboost".

```python
import json
import joblib
from huggingface_hub import hf_hub_download

MODEL_ID = "faodl/agri-soybeans-classifier"
BASELINE = "logistic"

texts = [
    "Maize production forecasts were revised after delayed rains.",
    "The central bank published new exchange rate statistics.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}_tfidf.joblib",
)
report_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename="report.json",
)

pipeline = joblib.load(model_path)
with open(report_path, encoding="utf-8") as handle:
    report = json.load(handle)

threshold = next(
    result["validation_best_threshold"]["threshold"]
    for result in report["results"]
    if result["model_type"] == f"{BASELINE}_tfidf"
)

probabilities = pipeline.predict_proba(texts)[:, 1]
for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})
```

### Sentence-Embedding Baselines

Available embedding baseline names in this run: "embedding-logistic", "embedding-svm", "embedding-lightgbm".

```python
import joblib
from huggingface_hub import hf_hub_download
from sentence_transformers import SentenceTransformer

MODEL_ID = "faodl/agri-soybeans-classifier"
BASELINE = "embedding-logistic"

texts = [
    "Wheat export inspections rose as demand from importers increased.",
    "The sports ministry announced a new stadium renovation plan.",
]

model_path = hf_hub_download(
    repo_id=MODEL_ID,
    repo_type="model",
    filename=f"baselines/{BASELINE}/{BASELINE}.joblib",
)
artifact = joblib.load(model_path)
embedding_model = SentenceTransformer(artifact["embedding_model_name"])
embeddings = embedding_model.encode(
    texts,
    batch_size=artifact.get("embedding_batch_size", 64),
    convert_to_numpy=True,
    normalize_embeddings=artifact.get("normalize_embeddings", True),
)
probabilities = artifact["classifier"].predict_proba(embeddings)[:, 1]
threshold = artifact["validation_best_threshold"]["threshold"]

for text, probability in zip(texts, probabilities):
    label = "RELEVANT" if probability >= threshold else "NOT_RELEVANT"
    print({"text": text, "probability_positive": float(probability), "label": label})
```

## Files

- `REPORT.md`: Markdown report for this training run.
- `report.json`: Machine-readable report containing metrics and thresholds.
- `transformer/`: Fine-tuned Transformer artifacts, when Transformer training is enabled.
- `baselines/`: TF-IDF and sentence-embedding baseline artifacts, when baseline training is enabled.
- `*/validation_predictions.csv` and `*/test_predictions.csv`: Split-level predictions.