Update README.md

README.md

@@ -7,74 +7,122 @@ tags:
 model-index:
 - name: schedulebot-nlu-engine
   results: []
+datasets:
+- andreaceto/hasd
+language:
+- en
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
 
-# schedulebot-nlu-engine
+# Schedulebot-nlu-engine
 
-This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.4952
-- Intent Accuracy: 0.8767
-- Intent F1: 0.8762
-- Ner F1: 0.9222
+## Model Description
 
-## Model description
+This model is a multi-task Natural Language Understanding (NLU) engine designed specifically for an appointment scheduling chatbot. It is fine-tuned from a **`distilbert-base-uncased`** backbone and is capable of performing two tasks simultaneously:
 
-More information needed
+- **Intent Classification**: Identifying the user's primary goal (e.g., `schedule`, `cancel`).
+- **Named Entity Recognition (NER)**: Extracting custom, domain-specific entities (e.g., `appointment_type`).
 
-## Intended uses & limitations
+This model stands out due to its custom classification heads, which use a more complex architecture to improve performance on nuanced tasks.
 
-More information needed
+## Model Architecture
 
-## Training and evaluation data
+The model uses a standard `distilbert-base-uncased` model as its core feature extractor. Two custom classification "heads" are placed on top of this base to perform the downstream tasks.
 
-More information needed
+- **Base Model**: `distilbert-base-uncased`
+- **Classifier Heads**: each head is a Multi-Layer Perceptron (MLP) with the following structure to allow for more complex feature interpretation:
+    1. A Linear layer projecting the transformer's output dimension (768) to an intermediate size (384).
+    2. A GELU activation function.
+    3. A Dropout layer with a rate of 0.3 for regularization.
+    4. A final Linear layer projecting the intermediate size to the number of output labels for the specific task (intent or NER).
 
-## Training procedure
+## Intended Use
 
-### Training hyperparameters
+This model is intended to be the core NLU component of a conversational AI system for managing appointments. It takes raw user text as input and outputs a structured JSON object containing the predicted intent and a list of extracted entities.
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-06
-- train_batch_size: 32
-- eval_batch_size: 32
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- num_epochs: 20
+```python
+from transformers import AutoTokenizer
 
-### Training results
 
-| Training Loss | Epoch | Step | Validation Loss | Intent Accuracy | Intent F1 | Ner F1 |
-|:-------------:|:-----:|:----:|:---------------:|:---------------:|:---------:|:------:|
-| 0.9887        | 1.0   | 64   | 0.7098          | 0.8219          | 0.8214    | 0.9200 |
-| 1.0663        | 2.0   | 128  | 0.6923          | 0.8196          | 0.8187    | 0.9200 |
-| 0.9498        | 3.0   | 192  | 0.6631          | 0.8288          | 0.8267    | 0.9188 |
-| 0.8652        | 4.0   | 256  | 0.6372          | 0.8447          | 0.8432    | 0.9199 |
-| 0.8757        | 5.0   | 320  | 0.6231          | 0.8311          | 0.8312    | 0.9211 |
-| 0.8406        | 6.0   | 384  | 0.5914          | 0.8562          | 0.8559    | 0.9194 |
-| 0.8455        | 7.0   | 448  | 0.5781          | 0.8562          | 0.8551    | 0.9216 |
-| 0.7884        | 8.0   | 512  | 0.5670          | 0.8630          | 0.8619    | 0.9229 |
-| 0.8384        | 9.0   | 576  | 0.5606          | 0.8607          | 0.8604    | 0.9210 |
-| 0.7189        | 10.0  | 640  | 0.5468          | 0.8539          | 0.8530    | 0.9246 |
-| 0.7834        | 11.0  | 704  | 0.5350          | 0.8767          | 0.8765    | 0.9221 |
-| 0.774         | 12.0  | 768  | 0.5231          | 0.8699          | 0.8691    | 0.9222 |
-| 0.7204        | 13.0  | 832  | 0.5185          | 0.8767          | 0.8764    | 0.9216 |
-| 0.7161        | 14.0  | 896  | 0.5131          | 0.8790          | 0.8787    | 0.9216 |
-| 0.7283        | 15.0  | 960  | 0.5094          | 0.8721          | 0.8719    | 0.9205 |
-| 0.8251        | 16.0  | 1024 | 0.5029          | 0.8744          | 0.8739    | 0.9222 |
-| 0.6644        | 17.0  | 1088 | 0.4997          | 0.8790          | 0.8786    | 0.9228 |
-| 0.7437        | 18.0  | 1152 | 0.4966          | 0.8767          | 0.8762    | 0.9228 |
-| 0.6905        | 19.0  | 1216 | 0.4952          | 0.8767          | 0.8762    | 0.9222 |
-| 0.6605        | 20.0  | 1280 | 0.4952          | 0.8767          | 0.8762    | 0.9222 |
+```
 
+## Training Data
 
-### Framework versions
+The model was trained on the **HASD (Hybrid Appointment Scheduling Dataset)**, a custom dataset built specifically for this task.
 
-- Transformers 4.53.1
-- Pytorch 2.6.0+cu124
-- Datasets 4.0.0
-- Tokenizers 0.21.2
+- **Source**: The dataset is a hybrid of real-world conversational examples from `clinc/clinc_oos` (for simple intents) and synthetically generated, template-based examples for complex scheduling intents.
+- **Balancing**: To combat class imbalance, intents sourced from `clinc/clinc_oos` were **down-sampled** to a maximum of **150 examples** each. [cite: multitask_model.ipynb]
+- **Augmentation**: To increase data diversity for complex intents (`schedule`, `reschedule`, etc.), **Contextual Word Replacement** was used. A `distilbert-base-uncased` model augmented the templates by replacing non-placeholder words with contextually relevant synonyms. [cite: multitask_model.ipynb]
+
+### Intents
+
+The model is trained to recognize the following intents:
+`schedule`, `reschedule`, `cancel`, `query_avail`, `greeting`, `positive_reply`, `negative_reply`, `bye`, `oos` (out-of-scope). [cite: multitask_model.ipynb]
+
+### Entities
+
+The model is trained to recognize the following custom named entities:
+`practitioner_name`, `appointment_type`, `appointment_id`. [cite: multitask_model.ipynb]
+
+## Training Procedure
+
+The model was trained using a two-stage fine-tuning strategy to ensure stability and performance.
+
+### Stage 1: Training the Classifier Heads
+
+- The `distilbert-base-uncased` base model was **frozen**. [cite: multitask_model.ipynb]
+- Only the randomly initialized MLP heads for intent and NER classification were trained.
+- This was done for **5 epochs** with a higher learning rate (`5e-4`), allowing the new layers to learn the task basics without disrupting the pre-trained backbone. [cite: multitask_model.ipynb]
+
+### Stage 2: Selective Fine-Tuning
+
+- The classification heads were kept trainable, and the **top two layers** of the DistilBERT backbone were **unfrozen**. [cite: multitask_model.ipynb]
+- The entire model was then fine-tuned for **3 epochs** with a much lower learning rate (`2e-5`). [cite: multitask_model.ipynb]
+- This gradual unfreezing approach allows the model to adapt its most task-specific layers to the new data while preserving the powerful, general-purpose knowledge in the lower layers.
+
+## Evaluation
+
+The model was evaluated on a held-out test set, and its performance was measured for both tasks.
+
+### Intent Classification Performance
+
+| Intent | Precision | Recall | F1-Score | Support |
+| --- | --- | --- | --- | --- |
+| bye | 1.00 | 1.00 | 1.00 | 22 |
+| cancel | 1.00 | 0.95 | 0.98 | 21 |
+| greeting | 1.00 | 1.00 | 1.00 | 23 |
+| negative_reply | 0.96 | 1.00 | 0.98 | 22 |
+| oos | 1.00 | 1.00 | 1.00 | 22 |
+| positive_reply | 1.00 | 0.96 | 0.98 | 23 |
+| query_avail | 0.95 | 1.00 | 0.98 | 21 |
+| reschedule | 0.96 | 1.00 | 0.98 | 22 |
+| schedule | 0.95 | 0.95 | 0.95 | 21 |
+| **Accuracy** |  |  | **0.98** | **197** |
+| **Macro Avg** | **0.98** | **0.98** | **0.98** | **197** |
+| **Weighted Avg** | **0.98** | **0.98** | **0.98** | **197** |
+
+*[Based on the classification report in the provided `multitask_model.ipynb` notebook.]* [cite: multitask_model.ipynb]
+
+### NER (Token Classification) Performance
+
+| Entity | Precision | Recall | F1-Score | Support |
+| --- | --- | --- | --- | --- |
+| B-appointment_id | 1.00 | 1.00 | 1.00 | 25 |
+| B-appointment_type | 1.00 | 1.00 | 1.00 | 33 |
+| B-practitioner_name | 1.00 | 1.00 | 1.00 | 44 |
+| O | 1.00 | 1.00 | 1.00 | 1342 |
+| **Micro Avg** | **1.00** | **1.00** | **1.00** | 1444 |
+| **Macro Avg** | **1.00** | **1.00** | **1.00** | 1444 |
+| **Weighted Avg** | **1.00** | **1.00** | **1.00** | 1444 |
+
+*[Based on the classification report in the provided `multitask_model.ipynb` notebook.]* [cite: multitask_model.ipynb]
+
+The model achieves near-perfect results on the NER task and excellent results on the intent classification task for this specific dataset.
+
+## Limitations and Bias
+
+- The model's performance is highly dependent on the quality and scope of the **HASD dataset**. It may not generalize well to phrasing or appointment types significantly different from what it was trained on.
+- The dataset was primarily generated from templates, which may not capture the full diversity of real human language.
+- The model inherits any biases present in the `distilbert-base-uncased` model and the `clinc/clinc_oos` dataset.