README.md

---
license: apache-2.0
language:
  - en
base_model: distilbert-base-uncased
pipeline_tag: text-classification
tags:
  - text-classification
  - distilbert
  - onet
  - work-taxonomy
metrics:
  - accuracy
  - f1
---

# Task → AI Capability Classifier

This is a small AI model that reads a description of a work task and sorts it into one of nine
categories, based on the kind of thinking the task involves. For example, "reconcile invoices against
purchase orders" is mostly about *matching* records, while "draft a quarterly report" is about
*generating* new content.

It learned to do this from 18,796 real work tasks taken from O*NET, a public U.S. Department of Labor
database. Each of those tasks had already been labeled by hand with its main category, and the model
studied those examples until it could label new tasks on its own.

## The nine categories

| Category | What the task is mostly doing |
|----------|-------------------------------|
| INPUT | Entering or updating data (e.g. posting a journal entry) |
| EXTRACT | Pulling information out of documents (e.g. reading an invoice) |
| CLASSIFY | Sorting things into groups (e.g. routing a support ticket) |
| MATCH | Finding things that correspond (e.g. reconciling accounts) |
| DETECT | Spotting something unusual (e.g. flagging fraud) |
| GENERATE | Creating new content (e.g. writing a report) |
| ORCHESTRATE | Running a multi-step process end to end (e.g. procure-to-pay) |
| PREDICT | Forecasting from past patterns (e.g. demand forecasting) |
| CONVERSE | Talking with people to resolve something (e.g. customer support) |

## What it's for

Give it a task, and it tells you which category that task fits best. This is handy for getting a quick
read on what kind of work a role or a team is made up of. It's a useful signal, not a final answer —
it's a machine's best guess at a system of categories that people defined, so treat it as a helpful
second opinion rather than the source of truth.

## How to use it

```python
from transformers import pipeline
classifier = pipeline("text-classification", model="abandekar-dev/onet-capability-classifier")
classifier("Reconcile vendor invoices against purchase orders and flag discrepancies")
```

## What it learned from

- **Where the data came from:** 18,796 work tasks from O*NET, each already labeled with its main category.
- **How the data was divided:** The examples were split into three groups — one large group for the model
  to learn from, and two smaller groups held back to test it fairly on tasks it had never seen. The split
  was done carefully so that even the rarest category showed up in all three groups (otherwise the rare
  ones could have been left out of testing entirely, making the results meaningless).

One thing shaped almost everything about how this model behaves: **the categories are very unevenly
sized.** Some are common, some are rare.

| Category | Share of all tasks |
|----------|--------------------|
| ORCHESTRATE | 34.8% |
| CONVERSE | 16.9% |
| GENERATE | 13.9% |
| DETECT | 10.2% |
| EXTRACT | 7.9% |
| PREDICT | 6.1% |
| INPUT | 5.0% |
| CLASSIFY | 3.9% |
| MATCH | 1.3% |

ORCHESTRATE is more than a third of everything; MATCH is barely one in a hundred. That imbalance is the
key to reading the results below.

## How well it does

Two scores tell the story, and the gap between them is the important part:

| Score | Value | What it means |
|-------|-------|---------------|
| Accuracy | 78% | Out of all tasks, how many it labeled correctly overall |
| Balanced score (F1-macro) | 72% | How well it does when every category counts equally |

**Why two scores instead of one?** Accuracy alone is misleading here. Because ORCHESTRATE is so common,
a model could look decent just by being good at that one big category while being bad at the small ones.
The second score fixes that by treating all nine categories as equally important — so being weak on a rare
category actually shows up. That's why the balanced score is lower, and it's the one to trust. This model
was chosen using that fairer score.

Here's how it does category by category. The pattern is simple: the more examples a category had, the
better the model handles it.

| Category | How often it catches them | Number tested |
|----------|---------------------------|---------------|
| ORCHESTRATE | 87% | 655 |
| CONVERSE | 81% | 317 |
| GENERATE | 77% | 260 |
| EXTRACT | 76% | 150 |
| DETECT | 70% | 192 |
| INPUT | 70% | 94 |
| PREDICT | 68% | 115 |
| CLASSIFY | 63% | 73 |
| MATCH | 46% | 24 |

The common categories are reliable. The rarest one, MATCH, gets caught less than half the time — simply
because the model saw so few examples of it.

## A fix I tried, and chose not to keep

There's a standard technique for the imbalance problem: tell the model to treat mistakes on rare
categories as more costly, so it pays them more attention. I tried it. The result is a good lesson in
why the obvious fix isn't always the right one.

It did exactly what it's supposed to: it nearly doubled how often the model caught the rare MATCH tasks.
**But** it also got sloppier — it started over-guessing the rare categories, so more of those guesses were
wrong, and it got noticeably worse at the big common category it used to handle easily. Adding it up, the
fair score actually went *down* slightly.

Two reasons it didn't pay off. First, the imbalance here was only moderate, and the careful data split had
already handled much of it. Second — and more interesting — some of the model's mistakes aren't about rare
categories at all; they're because certain categories genuinely look alike. MATCH, EXTRACT, and INPUT all
involve handling data, so they're honestly easy to confuse even for a person reading the task. No amount of
the fix can teach a difference that isn't really there in the words.

So I kept the simpler version. That was a deliberate decision, not something I overlooked.

## Where it falls short

- **It's unreliable on rare categories,** especially MATCH. Don't lean on it there.
- **It only picks one category.** Real tasks often combine several; it only reports the strongest.
- **It knows O*NET-style wording.** Tasks phrased very differently may trip it up.
- **It's an approximation.** It's a machine's learned imitation of categories that people defined — useful
  for exploring and for double-checking the human labels, but not an authority over them.