license: apache-2.0
language:
- en
base_model: distilbert-base-uncased
pipeline_tag: text-classification
tags:
- text-classification
- distilbert
- onet
- work-taxonomy
metrics:
- accuracy
- f1
Task → AI Capability Classifier
This is a small AI model that reads a description of a work task and sorts it into one of nine categories, based on the kind of thinking the task involves. For example, "reconcile invoices against purchase orders" is mostly about matching records, while "draft a quarterly report" is about generating new content.
It learned to do this from 18,796 real work tasks taken from O*NET, a public U.S. Department of Labor database. Each of those tasks had already been labeled by hand with its main category, and the model studied those examples until it could label new tasks on its own.
The nine categories
| Category | What the task is mostly doing |
|---|---|
| INPUT | Entering or updating data (e.g. posting a journal entry) |
| EXTRACT | Pulling information out of documents (e.g. reading an invoice) |
| CLASSIFY | Sorting things into groups (e.g. routing a support ticket) |
| MATCH | Finding things that correspond (e.g. reconciling accounts) |
| DETECT | Spotting something unusual (e.g. flagging fraud) |
| GENERATE | Creating new content (e.g. writing a report) |
| ORCHESTRATE | Running a multi-step process end to end (e.g. procure-to-pay) |
| PREDICT | Forecasting from past patterns (e.g. demand forecasting) |
| CONVERSE | Talking with people to resolve something (e.g. customer support) |
What it's for
Give it a task, and it tells you which category that task fits best. This is handy for getting a quick read on what kind of work a role or a team is made up of. It's a useful signal, not a final answer — it's a machine's best guess at a system of categories that people defined, so treat it as a helpful second opinion rather than the source of truth.
How to use it
from transformers import pipeline
classifier = pipeline("text-classification", model="abandekar-dev/onet-capability-classifier")
classifier("Reconcile vendor invoices against purchase orders and flag discrepancies")
What it learned from
- Where the data came from: 18,796 work tasks from O*NET, each already labeled with its main category.
- How the data was divided: The examples were split into three groups — one large group for the model to learn from, and two smaller groups held back to test it fairly on tasks it had never seen. The split was done carefully so that even the rarest category showed up in all three groups (otherwise the rare ones could have been left out of testing entirely, making the results meaningless).
One thing shaped almost everything about how this model behaves: the categories are very unevenly sized. Some are common, some are rare.
| Category | Share of all tasks |
|---|---|
| ORCHESTRATE | 34.8% |
| CONVERSE | 16.9% |
| GENERATE | 13.9% |
| DETECT | 10.2% |
| EXTRACT | 7.9% |
| PREDICT | 6.1% |
| INPUT | 5.0% |
| CLASSIFY | 3.9% |
| MATCH | 1.3% |
ORCHESTRATE is more than a third of everything; MATCH is barely one in a hundred. That imbalance is the key to reading the results below.
How well it does
Two scores tell the story, and the gap between them is the important part:
| Score | Value | What it means |
|---|---|---|
| Accuracy | 78% | Out of all tasks, how many it labeled correctly overall |
| Balanced score (F1-macro) | 72% | How well it does when every category counts equally |
Why two scores instead of one? Accuracy alone is misleading here. Because ORCHESTRATE is so common, a model could look decent just by being good at that one big category while being bad at the small ones. The second score fixes that by treating all nine categories as equally important — so being weak on a rare category actually shows up. That's why the balanced score is lower, and it's the one to trust. This model was chosen using that fairer score.
Here's how it does category by category. The pattern is simple: the more examples a category had, the better the model handles it.
| Category | How often it catches them | Number tested |
|---|---|---|
| ORCHESTRATE | 87% | 655 |
| CONVERSE | 81% | 317 |
| GENERATE | 77% | 260 |
| EXTRACT | 76% | 150 |
| DETECT | 70% | 192 |
| INPUT | 70% | 94 |
| PREDICT | 68% | 115 |
| CLASSIFY | 63% | 73 |
| MATCH | 46% | 24 |
The common categories are reliable. The rarest one, MATCH, gets caught less than half the time — simply because the model saw so few examples of it.
A fix I tried, and chose not to keep
There's a standard technique for the imbalance problem: tell the model to treat mistakes on rare categories as more costly, so it pays them more attention. I tried it. The result is a good lesson in why the obvious fix isn't always the right one.
It did exactly what it's supposed to: it nearly doubled how often the model caught the rare MATCH tasks. But it also got sloppier — it started over-guessing the rare categories, so more of those guesses were wrong, and it got noticeably worse at the big common category it used to handle easily. Adding it up, the fair score actually went down slightly.
Two reasons it didn't pay off. First, the imbalance here was only moderate, and the careful data split had already handled much of it. Second — and more interesting — some of the model's mistakes aren't about rare categories at all; they're because certain categories genuinely look alike. MATCH, EXTRACT, and INPUT all involve handling data, so they're honestly easy to confuse even for a person reading the task. No amount of the fix can teach a difference that isn't really there in the words.
So I kept the simpler version. That was a deliberate decision, not something I overlooked.
Where it falls short
- It's unreliable on rare categories, especially MATCH. Don't lean on it there.
- It only picks one category. Real tasks often combine several; it only reports the strongest.
- It knows O*NET-style wording. Tasks phrased very differently may trip it up.
- It's an approximation. It's a machine's learned imitation of categories that people defined — useful for exploring and for double-checking the human labels, but not an authority over them.