| --- |
| language: en |
| license: mit |
| library_name: transformers |
| pipeline_tag: text-classification |
| base_model: distilbert-base-uncased |
| datasets: |
| - albertmartinez/OSDG |
| tags: |
| - sdg |
| - sustainable-development-goals |
| - policy |
| - text-classification |
| metrics: |
| - accuracy |
| - f1 |
| --- |
| |
| # sdg-classifier |
|
|
| A 16-class text classifier that assigns a passage to one UN Sustainable Development |
| Goal. Fine-tuned from `distilbert-base-uncased` on the OSDG Community Dataset. |
|
|
| Built primarily as an investigation into the benchmark itself. **Read the |
| limitations section before using the per-class scores to compare against other |
| models** — one class in this dataset is separable by writing style rather than |
| topic, which inflates aggregate metrics. |
|
|
| ## Intended use |
|
|
| - First-pass SDG tagging of policy text, dataset descriptions, and report excerpts, |
| for human review |
| - A reproducible baseline for OSDG-CD experiments |
|
|
| ## Out of scope |
|
|
| - Any use where the output is acted on without review |
| - SDG 17 (Partnerships), which is absent from the training data |
| - Detecting that a text is *not* SDG-related — there is no negative class, so the |
| model will assign a goal to any input, including unrelated text |
| - Short inputs. Training excerpts average ~100 words; performance on titles or |
| single phrases is untested |
|
|
| ## Results |
|
|
| Test set: 8,605 held-out examples. Training set filtered to agreement ≥ 0.5. |
|
|
| | Configuration | Accuracy | Macro F1 | |
| |---|---|---| |
| | 3 epochs | 0.793 | 0.762 | |
| | **6 epochs (this model)** | **0.799** | **0.772** | |
|
|
| Per-class (6 epochs): |
|
|
| | Class | Precision | Recall | F1 | Support | |
| |---|---|---|---|---| |
| | SDG 1 — No Poverty | 0.729 | 0.619 | 0.670 | 494 | |
| | SDG 2 — Zero Hunger | 0.777 | 0.800 | 0.789 | 506 | |
| | SDG 3 — Good Health | 0.904 | 0.861 | 0.882 | 526 | |
| | SDG 4 — Quality Education | 0.887 | 0.829 | 0.857 | 736 | |
| | SDG 5 — Gender Equality | 0.814 | 0.848 | 0.831 | 875 | |
| | SDG 6 — Clean Water | 0.764 | 0.807 | 0.785 | 549 | |
| | SDG 7 — Affordable Energy | 0.785 | 0.838 | 0.810 | 610 | |
| | SDG 8 — Decent Work | 0.399 | 0.532 | 0.456 | 284 | |
| | SDG 9 — Industry & Infrastructure | 0.782 | 0.753 | 0.767 | 543 | |
| | SDG 10 — Reduced Inequalities | 0.613 | 0.648 | 0.630 | 457 | |
| | SDG 11 — Sustainable Cities | 0.743 | 0.793 | 0.767 | 497 | |
| | SDG 12 — Responsible Consumption | 0.755 | 0.687 | 0.719 | 233 | |
| | SDG 13 — Climate Action | 0.779 | 0.726 | 0.751 | 452 | |
| | SDG 14 — Life Below Water | 0.871 | 0.792 | 0.830 | 231 | |
| | SDG 15 — Life on Land | 0.851 | 0.794 | 0.822 | 554 | |
| | SDG 16 — Peace & Justice | 0.984 | 0.985 | 0.984 | 1058 | |
|
|
| ## Limitations |
|
|
| **SDG 16's score is partly an artefact.** At 0.984 F1 it is far above every other |
| class, on arguably the framework's most abstract goal. A TF-IDF + logistic |
| regression model trained on only the **first eight words** of each excerpt reaches |
| 0.739 F1 on SDG 16 against a macro average of 0.327 — eight words carry no topical |
| content, so the class is identifiable from stylistic register alone. Sampled SDG 16 |
| excerpts are overwhelmingly academic law and political-science abstracts |
| ("This article provides a theoretically informed analysis…"). The signal is |
| distributed through the text, not confined to the opening: removing the lead |
| sentence cost SDG 16 only 0.016 F1 while costing other classes 0.05–0.10. |
|
|
| SDG 16 is 12.3% of the test set, so this inflates the aggregate numbers above. |
| Per-class scores on this benchmark are not comparable across goals. |
|
|
| **SDG 8 is unreliable.** Precision 0.399 — when this model predicts SDG 8, it is |
| wrong roughly 6 times in 10. SDG 1, 8, and 10 form a confusion cluster |
| (1→10 at 0.16, 8→10 at 0.12, 10→8 at 0.11) reflecting genuine conceptual overlap |
| between poverty, decent work, and inequality. Treat SDG 8 predictions as low |
| confidence. |
|
|
| **Single-label on a multi-label problem.** Real passages often address several |
| goals at once. This model returns one. |
|
|
| **No confidence intervals.** Single seed, single run. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Base model | `distilbert-base-uncased` | |
| | Dataset | `albertmartinez/OSDG`, config `2024-04-01` | |
| | Filter | agreement ≥ 0.5 | |
| | Splits | 30,978 train / 3,442 val / 8,605 test | |
| | Max length | 256 tokens | |
| | Epochs | 6 | |
| | Batch size | 32 | |
| | Learning rate | 3e-5, 10% warmup | |
| | Hardware | 1 Kaggle session, T4 | |
| | Model selection | best val macro-F1 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import pipeline |
| |
| clf = pipeline("text-classification", model="USERNAME/sdg-classifier") |
| clf("Rural households gained access to piped drinking water and sanitation.") |
| # [{'label': 'SDG 6', 'score': 0.59}] |
| ``` |
|
|
| ## Citation |
|
|
| Dataset: OSDG, UNDP IICPSD SDG AI Lab, & PPMI. *OSDG Community Dataset (OSDG-CD)*. |
| Zenodo. https://doi.org/10.5281/zenodo.5550238 (CC BY 4.0) |
