Create README.md

README.md

@@ -0,0 +1,103 @@
+# Skill Filtering BERT - Fine-tuned for Online Job Advertisements (OJAs)
+
+## Model Overview
+**Skill Filtering BERT** is a fine-tuned BERT-based model designed for the **information filtering task** of identifying sentences related to **skills** in **Online Job Advertisements (OJAs)**. The model automates the extraction of relevant information, reducing noise and processing complexity in scraped job advertisements by classifying each sentence as skill-relevant or not.
+
+---
+
+## Background
+Information filtering systems automate the extraction of relevant information to handle large information flows and mitigate overload, as described in *Hanani et al. (2001)*. Online Job Advertisements (OJAs) often include extraneous elements, such as web page descriptions, layout strings, or menu options, introduced during the scraping process. This noise necessitates a **cleaning step**, which we treat as an **information filtering task**.
+
+Given an OJA represented as a set of \(n\) sentences:
+
+OJA = {f_1, f_2, ..., f_n}
+
+the filtering step produces a **filtered set of \(m\) sentences** (\(m \leq n\)) that are skill-relevant:
+
+FilteredOJA = {c_1, c_2, ..., c_m}
+
+This model uses a fine-tuned BERT to accomplish this filtering, improving efficiency in downstream skill extraction tasks.
+
+---
+
+## Training Process
+
+The model was fine-tuned in two stages:
+
+### Stage 1: Initial Fine-Tuning
+1. **Dataset:**  
+   The ESCO taxonomy was used to construct a dataset of ~25,000 sentences, comprising a balanced distribution of:
+   - **Skill-related sentences** (class 1)  
+   - **Occupation-related sentences** (class 0)  
+
+   ESCO was chosen because its skill descriptions closely resemble the contexts in which skills appear in OJAs. By training BERT on these descriptions, the model learns to differentiate between skills and occupations based on contextual clues.
+
+2. **Training Details:**  
+   - **Training Dataset:** 80% of rows  
+   - **Validation Dataset:** 20% of rows  
+   - **Loss Function:** Cross-entropy  
+   - **Batch Size:** 16  
+   - **Epochs:** 4  
+
+3. **Results:**  
+   - **Training Loss:** 0.0211  
+   - **Precision:** 89%  
+   - **Recall:** 94%  
+
+4. **Evaluation:**  
+   On a manually labeled dataset of 400 OJAs (split into sentences):
+   - **Precision:** 40%  
+   - **Recall:** 81%  
+
+---
+
+### Stage 2: Second Fine-Tuning
+1. **Dataset:**  
+   To improve recall and precision, we manually labeled **300 OJAs** (split into sentences). Sentences were annotated as:
+   - **Skill-relevant (class 1)**  
+   - **Non-skill-relevant (class 0)**  
+
+   To emphasize skill-related sentences, a **cost matrix** was introduced, doubling the weight for class 1.
+
+2. **Training Details:**  
+   - **Training Dataset:** 75% of manually labeled OJAs  
+   - **Validation Dataset:** 25% of manually labeled OJAs  
+   - **Batch Size:** 16  
+   - **Epochs:** 4  
+
+3. **Results:**  
+   - **Precision:** 71%  
+   - **Recall:** 93%  
+
+4. **Final Evaluation:**  
+   Evaluated on the remaining 100 manually labeled OJAs, the model demonstrated significant improvements in identifying skill-relevant sentences.
+
+---
+
+## Model Usage
+
+This model is ideal for organizations and researchers working on **labour market analysis**, **skill extraction**, or similar NLP tasks requiring fine-grained sentence filtering. By processing OJAs to identify skill-relevant sentences, downstream tasks like taxonomy mapping or skill prediction can be performed with higher precision and reduced noise.
+
+### How to Use the Model
+
+You can load the model using the Hugging Face Transformers library as follows:
+
+```python
+from transformers import BertForSequenceClassification, BertTokenizer
+
+# Load the model and tokenizer
+model_name = "username/skill-filtering-bert"
+model = BertForSequenceClassification.from_pretrained(model_name)
+tokenizer = BertTokenizer.from_pretrained(model_name)
+
+# Example input: a single sentence
+sentence = "This job requires proficiency in Python programming."
+inputs = tokenizer(sentence, return_tensors="pt")
+
+# Get predictions
+outputs = model(**inputs)
+logits = outputs.logits
+predicted_class = logits.argmax().item()
+
+# Class 1 = Skill-relevant, Class 0 = Non-skill-relevant
+print(f"Predicted Class: {predicted_class}")