# AMIS Commodity Classifier Training Report

- Dataset: `faodl/amis-agri-wheat`
- Dataset subset: ``
- Dataset revision: `main`
- Text column: `chunk_text`
- Label column: `label`
- Transformer: `FacebookAI/xlm-roberta-base`
- Generated at: `2026-05-29T18:13:08.384805+00:00`

## Dataset Summary

| Split | Rows | Label 0 | Label 1 | Unique groups | Mean text length |
| --- | ---: | ---: | ---: | ---: | ---: |
| train | 3622 | 2163 | 1459 | 1850 | 644.8 |
| validation | 759 | 486 | 273 | 396 | 636.7 |
| test | 762 | 470 | 292 | 397 | 643.3 |

## Threshold Comparison on Validation Split

Validation metrics document threshold selection and tuning behavior; test metrics remain the primary estimate of out-of-sample performance.

| Model | Threshold | Accuracy | Precision | Recall | F1 | ROC AUC | Average precision |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| logistic_tfidf | 0.500 | 0.818 | 0.718 | 0.813 | 0.763 | 0.907 | 0.867 |
| logistic_tfidf | 0.470 | 0.823 | 0.709 | 0.864 | 0.779 | 0.907 | 0.867 |
| xgboost_tfidf | 0.500 | 0.868 | 0.808 | 0.832 | 0.819 | 0.935 | 0.892 |
| xgboost_tfidf | 0.520 | 0.871 | 0.816 | 0.828 | 0.822 | 0.935 | 0.892 |
| embedding-logistic_sentence_embeddings | 0.500 | 0.783 | 0.658 | 0.824 | 0.732 | 0.862 | 0.780 |
| embedding-logistic_sentence_embeddings | 0.521 | 0.791 | 0.673 | 0.813 | 0.736 | 0.862 | 0.780 |
| embedding-svm_sentence_embeddings | 0.500 | 0.804 | 0.714 | 0.758 | 0.735 | 0.869 | 0.792 |
| embedding-svm_sentence_embeddings | 0.473 | 0.805 | 0.704 | 0.791 | 0.745 | 0.869 | 0.792 |
| embedding-lightgbm_sentence_embeddings | 0.500 | 0.791 | 0.694 | 0.747 | 0.720 | 0.868 | 0.786 |
| embedding-lightgbm_sentence_embeddings | 0.433 | 0.800 | 0.693 | 0.795 | 0.741 | 0.868 | 0.786 |
| transformer | 0.500 | 0.925 | 0.894 | 0.897 | 0.896 | 0.956 | 0.914 |
| transformer | 0.203 | 0.926 | 0.883 | 0.916 | 0.899 | 0.956 | 0.914 |

## Threshold Comparison on Test Split

| Model | Threshold | Accuracy | Precision | Recall | F1 | ROC AUC | Average precision |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| logistic_tfidf | 0.500 | 0.803 | 0.715 | 0.808 | 0.759 | 0.888 | 0.827 |
| logistic_tfidf | 0.470 | 0.797 | 0.688 | 0.860 | 0.764 | 0.888 | 0.827 |
| xgboost_tfidf | 0.500 | 0.835 | 0.773 | 0.805 | 0.789 | 0.910 | 0.831 |
| xgboost_tfidf | 0.520 | 0.835 | 0.777 | 0.798 | 0.787 | 0.910 | 0.831 |
| embedding-logistic_sentence_embeddings | 0.500 | 0.782 | 0.699 | 0.757 | 0.727 | 0.877 | 0.821 |
| embedding-logistic_sentence_embeddings | 0.521 | 0.789 | 0.713 | 0.750 | 0.731 | 0.877 | 0.821 |
| embedding-svm_sentence_embeddings | 0.500 | 0.818 | 0.778 | 0.733 | 0.755 | 0.883 | 0.824 |
| embedding-svm_sentence_embeddings | 0.473 | 0.812 | 0.758 | 0.750 | 0.754 | 0.883 | 0.824 |
| embedding-lightgbm_sentence_embeddings | 0.500 | 0.798 | 0.740 | 0.729 | 0.734 | 0.892 | 0.847 |
| embedding-lightgbm_sentence_embeddings | 0.433 | 0.806 | 0.735 | 0.771 | 0.753 | 0.892 | 0.847 |
| transformer | 0.500 | 0.885 | 0.862 | 0.832 | 0.847 | 0.943 | 0.915 |
| transformer | 0.203 | 0.890 | 0.854 | 0.860 | 0.857 | 0.943 | 0.915 |

## Confusion Matrices on Test Split

Rows are true labels and columns are predicted labels.

### logistic_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 376 | 94 |
| RELEVANT | 56 | 236 |

### logistic_tfidf at threshold 0.470

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 356 | 114 |
| RELEVANT | 41 | 251 |

### xgboost_tfidf at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 401 | 69 |
| RELEVANT | 57 | 235 |

### xgboost_tfidf at threshold 0.520

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 403 | 67 |
| RELEVANT | 59 | 233 |

### embedding-logistic_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 375 | 95 |
| RELEVANT | 71 | 221 |

### embedding-logistic_sentence_embeddings at threshold 0.521

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 382 | 88 |
| RELEVANT | 73 | 219 |

### embedding-svm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 409 | 61 |
| RELEVANT | 78 | 214 |

### embedding-svm_sentence_embeddings at threshold 0.473

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 400 | 70 |
| RELEVANT | 73 | 219 |

### embedding-lightgbm_sentence_embeddings at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 395 | 75 |
| RELEVANT | 79 | 213 |

### embedding-lightgbm_sentence_embeddings at threshold 0.433

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 389 | 81 |
| RELEVANT | 67 | 225 |

### transformer at threshold 0.500

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 431 | 39 |
| RELEVANT | 49 | 243 |

### transformer at threshold 0.203

| True / Predicted | NOT_RELEVANT | RELEVANT |
| --- | ---: | ---: |
| NOT_RELEVANT | 427 | 43 |
| RELEVANT | 41 | 251 |


## Validation-Tuned Thresholds

- `logistic_tfidf`: threshold `0.470` (validation F1 `0.779`); test F1 change vs 0.5: `+0.005`.
- `xgboost_tfidf`: threshold `0.520` (validation F1 `0.822`); test F1 change vs 0.5: `-0.001`.
- `embedding-logistic_sentence_embeddings`: threshold `0.521` (validation F1 `0.736`); test F1 change vs 0.5: `+0.004`.
- `embedding-svm_sentence_embeddings`: threshold `0.473` (validation F1 `0.745`); test F1 change vs 0.5: `-0.001`.
- `embedding-lightgbm_sentence_embeddings`: threshold `0.433` (validation F1 `0.741`); test F1 change vs 0.5: `+0.018`.
- `transformer`: threshold `0.203` (validation F1 `0.899`); test F1 change vs 0.5: `+0.010`.

## Artifacts

- `logistic_tfidf`: `/content/agri-wheat-classifier/baselines/logistic`
- `xgboost_tfidf`: `/content/agri-wheat-classifier/baselines/xgboost`
- `embedding-logistic_sentence_embeddings`: `/content/agri-wheat-classifier/baselines/embedding-logistic`
- `embedding-svm_sentence_embeddings`: `/content/agri-wheat-classifier/baselines/embedding-svm`
- `embedding-lightgbm_sentence_embeddings`: `/content/agri-wheat-classifier/baselines/embedding-lightgbm`
- `transformer`: `/content/agri-wheat-classifier/transformer`