seed_data/onet_roles_curation_log.md

Role Curation Log

Generated 2026-04-19T09:25:05.374883+00:00Z by scripts/curate_roles.py

This log shows, per role, which skills were kept and which were dropped (with reason). Source of truth for keep decisions: seed_data/role_curation.yaml. Edit there + re-run.

Data Scientist

Kept (16 skills):

• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Pandas (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• NumPy (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Scikit-learn (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• PyTorch (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Statistical Hypothesis Testing (mandatory, weight=0.9, level=INTERMEDIATE, source=manual)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• TensorFlow (optional, weight=0.9, level=INTERMEDIATE, source=onet)
• AWS (optional, weight=0.8, level=INTERMEDIATE, source=onet)
• Tableau (optional, weight=0.8, level=INTERMEDIATE, source=onet)
• Apache Spark (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• Feature Engineering (optional, weight=0.7, level=INTERMEDIATE, source=manual)
• Microsoft Power BI (optional, weight=0.6, level=INTERMEDIATE, source=onet)
• Matplotlib (optional, weight=0.6, level=BEGINNER, source=manual)
• Docker (optional, weight=0.6, level=INTERMEDIATE, source=manual)

Dropped (13 skills):

• Apache Hadoop — Spark replaced it
• C++ — Low-level numerics only; not in PK DS postings
• Microsoft Azure — Pick one cloud — AWS retained as primary
• Microsoft Excel — R4: not core for DS (Excel is for DA/BI roles)
• Microsoft PowerPoint — Presentation tool, not measurable
• Java — Rare in DS workflow
• R — R4: PK DS market is Python+SQL; R rarely required (Glassdoor PK Data Scientist evidence)
• SAS — Enterprise legacy
• Snowflake — R4: warehouse skill — Data Engineer territory
• MATLAB — Academic only
• Jupyter Notebook — R4: assumed knowledge, not learnable on slider
• Seaborn — R4: covered by Matplotlib retention
• Data Structures & Algorithms — R4: interview prep, not posting requirement (per job-board evidence)

Data Analyst

Kept (10 skills):

• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Microsoft Excel (mandatory, weight=1.0, level=ADVANCED, source=onet)
• Microsoft Power BI (optional, weight=0.9, level=INTERMEDIATE, source=manual)
• Python (mandatory, weight=0.9, level=INTERMEDIATE, source=onet)
• Tableau (optional, weight=0.8, level=INTERMEDIATE, source=onet+manual)
• Pandas (optional, weight=0.7, level=INTERMEDIATE, source=manual)
• DAX (optional, weight=0.7, level=BEGINNER, source=manual)
• Power Query / M Language (optional, weight=0.6, level=BEGINNER, source=manual)
• Data Visualization (optional, weight=0.6, level=BEGINNER, source=manual)
• Statistical Hypothesis Testing (optional, weight=0.5, level=INTERMEDIATE, source=manual)

Dropped (7 skills):

• AWS — Junior DA in PK rarely owns cloud infra
• IBM SPSS Statistics — Enterprise legacy
• Microsoft Access — Legacy desktop DB
• Microsoft Office — Too generic
• Microsoft PowerPoint — Presentation tool
• R — R4: PK DA postings rarely list R; demoted then dropped
• SAS — Enterprise legacy; PK market is Python/SQL

Machine Learning Engineer

Kept (16 skills):

• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• PyTorch (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• Scikit-learn (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Docker (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• AWS (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Pandas (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• NumPy (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• TensorFlow (optional, weight=0.9, level=INTERMEDIATE, source=onet+manual)
• MLflow (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• Kubernetes (optional, weight=0.7, level=INTERMEDIATE, source=onet+manual)
• FastAPI (optional, weight=0.7, level=BEGINNER, source=manual)
• Apache Spark (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• Linux (optional, weight=0.6, level=INTERMEDIATE, source=onet)

Dropped (26 skills):

• Apache Hadoop — Spark replaced it
• C++ — Niche optimization only
• Microsoft Azure — Pick one cloud — AWS retained
• Microsoft Excel — Not core to ML eng
• Microsoft Power BI — BI tool
• Microsoft PowerPoint — Presentation tool
• Java — Rare in ML pipelines
• R — Statistics-heavy — DS territory
• SAS — Enterprise legacy
• Snowflake — Data Engineer territory
• Tableau — BI tool
• MATLAB — Academic only
• Ansible — DevOps territory
• Bash — R4: assumed background skill
• C — Legacy systems language
• Go — Rare in ML pipelines
• Terraform — DevOps territory
• JavaScript — Frontend, not ML
• Jenkins — Replaced by GitHub Actions; not ML-eng-specific
• NoSQL — Generic
• Scala — Spark+Scala niche, declining
• Splunk Enterprise — Observability niche
• ONNX — R4: niche optimization, not in PK ML postings
• Data Structures & Algorithms — R4: not in postings (interview prep)
• Object-Oriented Programming — R4: not in postings (interview prep)
• Agile / Scrum — R4: rarely in ML eng postings

Backend Developer

Kept (16 skills):

• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Django (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Node.js (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• PostgreSQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• REST API Design (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Docker (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• JWT / OAuth (mandatory, weight=0.8, level=BEGINNER, source=manual)
• FastAPI (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• TypeScript (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• AWS (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• NoSQL (optional, weight=0.6, level=INTERMEDIATE, source=onet)
• Laravel (optional, weight=0.4, level=BEGINNER, source=manual)
• PHP (optional, weight=0.4, level=BEGINNER, source=manual)

Dropped (34 skills):

• Apache Kafka — Data Engineer territory
• Jira — PM tool
• C — Not Python/Node-stack backend
• C# — Microsoft .NET stack — separate market
• C++ — Game/systems, not web backend
• CSS — Frontend
• XML — Legacy
• Angular — Frontend
• HTML — Frontend
• Terraform — DevOps territory
• JavaScript — no reason given (please add to role_curation.yaml drop_reasons)
• JSON — Assumed knowledge
• Jenkins — DevOps territory
• Kubernetes — R4: senior-only in PK; Docker covers entry-level
• Linux — R4: assumed background
• .NET — Microsoft enterprise stack — separate market
• Microsoft Azure — Pick one cloud — AWS retained
• Java — Spring stack — separate market
• React — Frontend
• Spring Boot — Java stack
• Spring Framework — Java stack
• Flask — R4: declining; FastAPI/Django dominate PK postings
• Redis — R4: rarely listed by name in PK postings (assumed within Postgres/MongoDB choice)
• pytest — R4: rarely listed by name in postings
• Celery — R4: rarely listed by name
• nginx — R4: rarely listed by name
• Pydantic — R4: rarely listed by name (implicit with FastAPI)
• Data Structures & Algorithms — R4: interview prep, not posting requirement
• Object-Oriented Programming — R4: assumed for interviews, not posting requirement
• Agile / Scrum — R4: not in posting requirement lists
• CI/CD Pipelines — R4: not as concept skill in postings
• GitHub Actions — R4: rare standalone in PK BE postings
• .NET / C# — R4: separate Pakistani market, deserves its own role tier
• Unit Testing — R4: not in posting requirement lists

Frontend Developer

Kept (14 skills):

• React (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• JavaScript (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• TypeScript (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• HTML (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• CSS (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• REST API Design (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Next.js (mandatory, weight=0.9, level=INTERMEDIATE, source=manual)
• Tailwind CSS (optional, weight=0.8, level=BEGINNER, source=manual)
• Redux (optional, weight=0.7, level=BEGINNER, source=manual)
• Node.js (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• Vite (optional, weight=0.5, level=BEGINNER, source=manual)
• TanStack Query (optional, weight=0.4, level=BEGINNER, source=manual)

Dropped (33 skills):

• AWS — DevOps territory
• Apache Kafka — Data Eng
• Jira — PM tool
• C# — Backend
• C++ — Not FE
• Docker — R4: rarely listed for FE-only roles
• Go — Not FE
• Angular — R4: separate role market in PK
• GraphQL — R4: rare in PK FE postings
• JSON — Assumed
• Jenkins — DevOps
• jQuery — Legacy
• Kubernetes — DevOps territory
• Linux — no reason given (please add to role_curation.yaml drop_reasons)
• Microsoft Azure — DevOps territory
• MongoDB — Backend/data
• MySQL — Backend/data
• NoSQL — Backend/data
• Java — Backend
• PHP — Backend (separate Full Stack stack)
• PostgreSQL — Backend/data
• Python — Backend
• Spring Boot — Java backend
• Spring Framework — Java backend
• SQL — Backend/data
• Vue.js — R4: separate role market in PK; React dominates
• WordPress — CMS — separate small-agency market
• Vitest — R4: testing rarely in PK FE postings
• Playwright — R4: testing rarely in PK FE postings
• Web Accessibility (a11y) — R4: rarely a posted requirement in PK
• Responsive Design — R4: assumed within React+Tailwind work
• Data Structures & Algorithms — R4: interview prep, not posting requirement
• Agile / Scrum — R4: not in FE posting requirement lists

Full Stack Developer

Kept (18 skills):

• JavaScript (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• TypeScript (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• React (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Node.js (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• PostgreSQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• REST API Design (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Docker (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• HTML (mandatory, weight=0.9, level=INTERMEDIATE, source=onet)
• CSS (mandatory, weight=0.9, level=INTERMEDIATE, source=onet)
• Next.js (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• Django (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• MongoDB (optional, weight=0.8, level=INTERMEDIATE, source=onet)
• Tailwind CSS (optional, weight=0.7, level=BEGINNER, source=manual)
• AWS (optional, weight=0.7, level=INTERMEDIATE, source=onet)

Dropped (34 skills):

• Apache Kafka — Data Eng
• Jira — PM tool
• C — Not web full-stack
• C# — .NET niche — separate stack
• C++ — Not web full-stack
• XML — Legacy
• Angular — Pick one FE framework — React retained
• Terraform — DevOps
• JSON — Assumed knowledge
• Jenkins — DevOps
• Kubernetes — no reason given (please add to role_curation.yaml drop_reasons)
• Linux — R4: assumed background
• .NET — .NET niche — separate stack
• Microsoft Azure — Pick one cloud — AWS retained
• NoSQL — Generic — MongoDB covers it
• Java — Spring niche — separate stack
• Spring Boot — Java stack
• Spring Framework — Java stack
• Go — Niche for full-stack web in PK
• GraphQL — R4: rare in PK FS postings
• jQuery — Legacy
• MySQL — PostgreSQL covers SQL needs
• PHP — R4: separate Laravel/PHP stack market — Backend covers
• Vue.js — Pick one FE framework — React retained
• WordPress — CMS — separate market
• Vite — R4: assumed within Next.js/React tooling
• Vitest — R4: testing rarely in PK postings
• JWT / OAuth — R4: covered by Backend role; FS focuses on stack
• Data Structures & Algorithms — R4: interview prep, not posting requirement
• Object-Oriented Programming — R4: not in posting requirement lists
• Agile / Scrum — R4: not in posting requirement lists
• CI/CD Pipelines — R4: not as concept skill in PK FS postings
• GitHub Actions — R4: rare standalone for FS
• Laravel — R4: separate Laravel/PHP stack market — Backend covers

Data Engineer

Kept (14 skills):

• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Apache Spark (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• Apache Airflow (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• ETL / ELT (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• AWS (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Snowflake (optional, weight=0.8, level=INTERMEDIATE, source=onet+manual)
• dbt (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• Apache Kafka (optional, weight=0.7, level=BEGINNER, source=manual)
• Google BigQuery (optional, weight=0.9, level=INTERMEDIATE, source=manual)
• Docker (optional, weight=0.7, level=INTERMEDIATE, source=manual)
• Microsoft Azure (optional, weight=0.6, level=INTERMEDIATE, source=onet)
• Data Modeling (optional, weight=0.6, level=INTERMEDIATE, source=manual)

Dropped (14 skills):

• Microsoft Access — Legacy desktop DB
• Microsoft Excel — R4: rare for DE postings
• Microsoft Office — Too generic
• Microsoft Outlook — Email client
• Microsoft Power BI — BI Analyst territory
• Microsoft PowerPoint — Presentation tool
• Tableau — BI Analyst territory
• Java — R4: declining for DE; Python+Scala dominate
• R — BI/stats territory; Python+SQL covers DE
• Delta Lake — R4: rare in PK entry-level DE postings
• Apache Iceberg — R4: rare in PK entry-level DE postings; Delta Lake (also dropped) was the safer pick but neither needed
• Databricks — R4: senior-only in PK postings
• Parquet — R4: rarely listed by name in postings (implicit with Spark/Airflow)
• Data Structures & Algorithms — R4: interview prep, not posting requirement

DevOps Engineer

Kept (16 skills):

• Docker (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• Kubernetes (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• Terraform (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• AWS (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• Linux (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Bash Scripting (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• CI/CD Pipelines (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Python (mandatory, weight=0.8, level=INTERMEDIATE, source=onet)
• GitHub Actions (optional, weight=0.8, level=INTERMEDIATE, source=manual)
• Microsoft Azure (optional, weight=0.7, level=BEGINNER, source=onet+manual)
• Ansible (optional, weight=0.7, level=BEGINNER, source=onet+manual)
• Prometheus (optional, weight=0.6, level=BEGINNER, source=manual)
• Grafana (optional, weight=0.6, level=BEGINNER, source=manual)
• Jenkins (optional, weight=0.5, level=INTERMEDIATE, source=onet+manual)

Dropped (16 skills):

• Amazon Web Services AWS CloudFormation — Terraform won
• Jira — R2: PM tool consistency
• Bash — Duplicate of Bash Scripting
• C — Not DevOps day-to-day
• C++ — Not DevOps day-to-day
• Go — K8s/Terraform internals only
• JavaScript — Not DevOps
• Microsoft Active Directory — Windows enterprise niche
• Microsoft PowerShell — Windows admin only
• Java — Rare in modern DevOps
• SQL — R4: debugging-only, not core skill
• Google Cloud Platform — R4: rare in PK DevOps postings (AWS/Azure dominate)
• Linux Administration — R2: duplicate of Linux
• ArgoCD — R4: senior-only in PK postings
• Helm — R4: senior-only in PK postings
• Computer Networking Fundamentals — R4: assumed background, not in posting requirement lists

AI Engineer (GenAI / LLM)

Kept (17 skills):

• Python (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• HuggingFace Transformers (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• LangChain (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• OpenAI API (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• RAG (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• Prompt Engineering (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• PyTorch (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Docker (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Git (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• GitHub (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• AWS (optional, weight=0.8, level=INTERMEDIATE, source=onet)
• Microsoft Azure (optional, weight=0.7, level=INTERMEDIATE, source=onet)
• Anthropic Claude API (optional, weight=0.7, level=BEGINNER, source=manual)
• pgvector (optional, weight=0.7, level=BEGINNER, source=manual)
• FastAPI (optional, weight=0.7, level=BEGINNER, source=manual)
• LangGraph (optional, weight=0.6, level=BEGINNER, source=manual)
• Sentence Transformers (optional, weight=0.6, level=BEGINNER, source=manual)

Dropped (26 skills):

• Ansible — DevOps
• Apache Hadoop — Big data — Data Eng
• Apache Spark — Big data — Data Eng
• Bash — Background
• C — Legacy
• C++ — Niche
• Go — Rare in AI eng
• Terraform — DevOps
• JavaScript — Not AI eng
• Jenkins — DevOps
• Kubernetes — R4: senior-only in PK AI eng postings
• Linux — Background
• NoSQL — Generic — vector DBs are the relevant DB type
• Java — Rare in AI eng
• R — Statistics, not LLM eng
• Scala — Big data, not LLM eng
• Splunk Enterprise — Observability niche
• SQL — R4: rare for AI Eng day-to-day
• TensorFlow — PyTorch dominates research/LLM
• LlamaIndex — R4: overlaps LangChain; rarely both required in PK postings
• Pinecone — R4: open-source vector DBs (pgvector, Chroma) winning
• LLM Evaluations — R4: concept rarely listed in postings
• LLM Observability — R4: concept rarely listed in postings
• LLM Fine-tuning — R4: rare in entry-level PK postings
• Object-Oriented Programming — R4: interview prep
• Data Structures & Algorithms — R4: interview prep

Business Intelligence Analyst

Kept (10 skills):

• Microsoft Power BI (mandatory, weight=1.0, level=INTERMEDIATE, source=onet+manual)
• DAX (mandatory, weight=1.0, level=INTERMEDIATE, source=manual)
• SQL (mandatory, weight=1.0, level=INTERMEDIATE, source=onet)
• Microsoft Excel (mandatory, weight=1.0, level=ADVANCED, source=onet)
• Power Query / M Language (mandatory, weight=0.9, level=BEGINNER, source=manual)
• Tableau (optional, weight=0.7, level=INTERMEDIATE, source=onet+manual)
• Snowflake (optional, weight=0.7, level=BEGINNER, source=manual)
• Python (optional, weight=0.6, level=INTERMEDIATE, source=onet)
• Star Schema Modeling (optional, weight=0.6, level=BEGINNER, source=manual)
• Data Storytelling (optional, weight=0.5, level=BEGINNER, source=manual)

Dropped (10 skills):

• AWS — BI Analyst consumes from cloud DWH, doesn't own infra
• Microsoft Azure — Same as AWS — BI Analyst doesn't own cloud infra
• Microsoft Office — Too generic — Microsoft Excel covers it
• Microsoft PowerPoint — Presentation, not BI engineering
• Oracle Cloud — Enterprise niche
• R — R4: PK BI market is Power BI+SQL+Excel; R rarely required
• SAP — ERP — niche
• SAS — Enterprise legacy
• Looker — R4: rare in PK BI postings (Power BI/Tableau dominate)
• dbt — R4: more for Data Eng; rarely in PK BI postings