---
language: en
license: mit
library_name: transformers
pipeline_tag: text-classification
base_model: distilbert-base-uncased
datasets:
  - albertmartinez/OSDG
tags:
  - sdg
  - sustainable-development-goals
  - policy
  - text-classification
metrics:
  - accuracy
  - f1
---

# sdg-classifier

A 16-class text classifier that assigns a passage to one UN Sustainable Development
Goal. Fine-tuned from `distilbert-base-uncased` on the OSDG Community Dataset.

Built primarily as an investigation into the benchmark itself. **Read the
limitations section before using the per-class scores to compare against other
models** — one class in this dataset is separable by writing style rather than
topic, which inflates aggregate metrics.

## Intended use

- First-pass SDG tagging of policy text, dataset descriptions, and report excerpts,
  for human review
- A reproducible baseline for OSDG-CD experiments

## Out of scope

- Any use where the output is acted on without review
- SDG 17 (Partnerships), which is absent from the training data
- Detecting that a text is *not* SDG-related — there is no negative class, so the
  model will assign a goal to any input, including unrelated text
- Short inputs. Training excerpts average ~100 words; performance on titles or
  single phrases is untested

## Results

Test set: 8,605 held-out examples. Training set filtered to agreement ≥ 0.5.

| Configuration | Accuracy | Macro F1 |
|---|---|---|
| 3 epochs | 0.793 | 0.762 |
| **6 epochs (this model)** | **0.799** | **0.772** |

Per-class (6 epochs):

| Class | Precision | Recall | F1 | Support |
|---|---|---|---|---|
| SDG 1 — No Poverty | 0.729 | 0.619 | 0.670 | 494 |
| SDG 2 — Zero Hunger | 0.777 | 0.800 | 0.789 | 506 |
| SDG 3 — Good Health | 0.904 | 0.861 | 0.882 | 526 |
| SDG 4 — Quality Education | 0.887 | 0.829 | 0.857 | 736 |
| SDG 5 — Gender Equality | 0.814 | 0.848 | 0.831 | 875 |
| SDG 6 — Clean Water | 0.764 | 0.807 | 0.785 | 549 |
| SDG 7 — Affordable Energy | 0.785 | 0.838 | 0.810 | 610 |
| SDG 8 — Decent Work | 0.399 | 0.532 | 0.456 | 284 |
| SDG 9 — Industry & Infrastructure | 0.782 | 0.753 | 0.767 | 543 |
| SDG 10 — Reduced Inequalities | 0.613 | 0.648 | 0.630 | 457 |
| SDG 11 — Sustainable Cities | 0.743 | 0.793 | 0.767 | 497 |
| SDG 12 — Responsible Consumption | 0.755 | 0.687 | 0.719 | 233 |
| SDG 13 — Climate Action | 0.779 | 0.726 | 0.751 | 452 |
| SDG 14 — Life Below Water | 0.871 | 0.792 | 0.830 | 231 |
| SDG 15 — Life on Land | 0.851 | 0.794 | 0.822 | 554 |
| SDG 16 — Peace & Justice | 0.984 | 0.985 | 0.984 | 1058 |

## Limitations

**SDG 16's score is partly an artefact.** At 0.984 F1 it is far above every other
class, on arguably the framework's most abstract goal. A TF-IDF + logistic
regression model trained on only the **first eight words** of each excerpt reaches
0.739 F1 on SDG 16 against a macro average of 0.327 — eight words carry no topical
content, so the class is identifiable from stylistic register alone. Sampled SDG 16
excerpts are overwhelmingly academic law and political-science abstracts
("This article provides a theoretically informed analysis…"). The signal is
distributed through the text, not confined to the opening: removing the lead
sentence cost SDG 16 only 0.016 F1 while costing other classes 0.05–0.10.

SDG 16 is 12.3% of the test set, so this inflates the aggregate numbers above.
Per-class scores on this benchmark are not comparable across goals.

**SDG 8 is unreliable.** Precision 0.399 — when this model predicts SDG 8, it is
wrong roughly 6 times in 10. SDG 1, 8, and 10 form a confusion cluster
(1→10 at 0.16, 8→10 at 0.12, 10→8 at 0.11) reflecting genuine conceptual overlap
between poverty, decent work, and inequality. Treat SDG 8 predictions as low
confidence.

**Single-label on a multi-label problem.** Real passages often address several
goals at once. This model returns one.

**No confidence intervals.** Single seed, single run.

## Training

| | |
|---|---|
| Base model | `distilbert-base-uncased` |
| Dataset | `albertmartinez/OSDG`, config `2024-04-01` |
| Filter | agreement ≥ 0.5 |
| Splits | 30,978 train / 3,442 val / 8,605 test |
| Max length | 256 tokens |
| Epochs | 6 |
| Batch size | 32 |
| Learning rate | 3e-5, 10% warmup |
| Hardware | 1 Kaggle session, T4 |
| Model selection | best val macro-F1 |

## Usage

```python
from transformers import pipeline

clf = pipeline("text-classification", model="USERNAME/sdg-classifier")
clf("Rural households gained access to piped drinking water and sanitation.")
# [{'label': 'SDG 6', 'score': 0.59}]
```

## Citation

Dataset: OSDG, UNDP IICPSD SDG AI Lab, & PPMI. *OSDG Community Dataset (OSDG-CD)*.
Zenodo. https://doi.org/10.5281/zenodo.5550238 (CC BY 4.0)