abandekar-dev
/

onet-capability-classifier

@@ -1,199 +1,141 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+license: apache-2.0
+language:
+  - en
+base_model: distilbert-base-uncased
+pipeline_tag: text-classification
+tags:
+  - text-classification
+  - distilbert
+  - onet
+  - work-taxonomy
+metrics:
+  - accuracy
+  - f1
 ---
+# Task → AI Capability Classifier
+This is a small AI model that reads a description of a work task and sorts it into one of nine
+categories, based on the kind of thinking the task involves. For example, "reconcile invoices against
+purchase orders" is mostly about *matching* records, while "draft a quarterly report" is about
+*generating* new content.
+It learned to do this from 18,796 real work tasks taken from O*NET, a public U.S. Department of Labor
+database. Each of those tasks had already been labeled by hand with its main category, and the model
+studied those examples until it could label new tasks on its own.
+## The nine categories
+| Category | What the task is mostly doing |
+|----------|-------------------------------|
+| INPUT | Entering or updating data (e.g. posting a journal entry) |
+| EXTRACT | Pulling information out of documents (e.g. reading an invoice) |
+| CLASSIFY | Sorting things into groups (e.g. routing a support ticket) |
+| MATCH | Finding things that correspond (e.g. reconciling accounts) |
+| DETECT | Spotting something unusual (e.g. flagging fraud) |
+| GENERATE | Creating new content (e.g. writing a report) |
+| ORCHESTRATE | Running a multi-step process end to end (e.g. procure-to-pay) |
+| PREDICT | Forecasting from past patterns (e.g. demand forecasting) |
+| CONVERSE | Talking with people to resolve something (e.g. customer support) |
+## What it's for
+Give it a task, and it tells you which category that task fits best. This is handy for getting a quick
+read on what kind of work a role or a team is made up of. It's a useful signal, not a final answer —
+it's a machine's best guess at a system of categories that people defined, so treat it as a helpful
+second opinion rather than the source of truth.
+## How to use it
+```python
+from transformers import pipeline
+classifier = pipeline("text-classification", model="abandekar-dev/onet-capability-classifier")
+classifier("Reconcile vendor invoices against purchase orders and flag discrepancies")
+```
+## What it learned from
+- **Where the data came from:** 18,796 work tasks from O*NET, each already labeled with its main category.
+- **How the data was divided:** The examples were split into three groups — one large group for the model
+  to learn from, and two smaller groups held back to test it fairly on tasks it had never seen. The split
+  was done carefully so that even the rarest category showed up in all three groups (otherwise the rare
+  ones could have been left out of testing entirely, making the results meaningless).
+One thing shaped almost everything about how this model behaves: **the categories are very unevenly
+sized.** Some are common, some are rare.
+| Category | Share of all tasks |
+|----------|--------------------|
+| ORCHESTRATE | 34.8% |
+| CONVERSE | 16.9% |
+| GENERATE | 13.9% |
+| DETECT | 10.2% |
+| EXTRACT | 7.9% |
+| PREDICT | 6.1% |
+| INPUT | 5.0% |
+| CLASSIFY | 3.9% |
+| MATCH | 1.3% |
+ORCHESTRATE is more than a third of everything; MATCH is barely one in a hundred. That imbalance is the
+key to reading the results below.
+## How well it does
+Two scores tell the story, and the gap between them is the important part:
+| Score | Value | What it means |
+|-------|-------|---------------|
+| Accuracy | 78% | Out of all tasks, how many it labeled correctly overall |
+| Balanced score (F1-macro) | 72% | How well it does when every category counts equally |
+**Why two scores instead of one?** Accuracy alone is misleading here. Because ORCHESTRATE is so common,
+a model could look decent just by being good at that one big category while being bad at the small ones.
+The second score fixes that by treating all nine categories as equally important — so being weak on a rare
+category actually shows up. That's why the balanced score is lower, and it's the one to trust. This model
+was chosen using that fairer score.
+Here's how it does category by category. The pattern is simple: the more examples a category had, the
+better the model handles it.
+| Category | How often it catches them | Number tested |
+|----------|---------------------------|---------------|
+| ORCHESTRATE | 87% | 655 |
+| CONVERSE | 81% | 317 |
+| GENERATE | 77% | 260 |
+| EXTRACT | 76% | 150 |
+| DETECT | 70% | 192 |
+| INPUT | 70% | 94 |
+| PREDICT | 68% | 115 |
+| CLASSIFY | 63% | 73 |
+| MATCH | 46% | 24 |
+The common categories are reliable. The rarest one, MATCH, gets caught less than half the time — simply
+because the model saw so few examples of it.
+## A fix I tried, and chose not to keep
+There's a standard technique for the imbalance problem: tell the model to treat mistakes on rare
+categories as more costly, so it pays them more attention. I tried it. The result is a good lesson in
+why the obvious fix isn't always the right one.
+It did exactly what it's supposed to: it nearly doubled how often the model caught the rare MATCH tasks.
+**But** it also got sloppier — it started over-guessing the rare categories, so more of those guesses were
+wrong, and it got noticeably worse at the big common category it used to handle easily. Adding it up, the
+fair score actually went *down* slightly.
+Two reasons it didn't pay off. First, the imbalance here was only moderate, and the careful data split had
+already handled much of it. Second — and more interesting — some of the model's mistakes aren't about rare
+categories at all; they're because certain categories genuinely look alike. MATCH, EXTRACT, and INPUT all
+involve handling data, so they're honestly easy to confuse even for a person reading the task. No amount of
+the fix can teach a difference that isn't really there in the words.
+So I kept the simpler version. That was a deliberate decision, not something I overlooked.
+## Where it falls short
+- **It's unreliable on rare categories,** especially MATCH. Don't lean on it there.
+- **It only picks one category.** Real tasks often combine several; it only reports the strongest.
+- **It knows O*NET-style wording.** Tasks phrased very differently may trip it up.
+- **It's an approximation.** It's a machine's learned imitation of categories that people defined — useful
+  for exploring and for double-checking the human labels, but not an authority over them.