Upload BERT-Text2Date.md

BERT-Text2Date.md

@@ -0,0 +1,149 @@
+
+
+# Model Card for BERT-Text2Date
+
+## Model Overview
+
+**Model Name:** BERT-Text2Date
+**Model Type:** BERT (Encoder-only architecture)  
+**Language:** Persian
+
+**Description:**  
+This model is designed to process and generate Persian dates in both formal (YYYY-MM-DD) and informal formats. It utilizes a dataset that includes various representations of dates, allowing for effective training in understanding and predicting Persian date formats.
+
+## Dataset
+
+**Dataset Description:**  
+The dataset consists of two types of dates: formal and informal. It is generated using two main functions:
+
+- **`convert_year_to_persian(year)`**: Converts years to Persian format, currently supporting the year 1400.
+- **`generate_date_mappings_with_persian_year(start_year, end_year)`**: Generates dates for a specified range, considering the number of days in each month.
+
+**Data Formats:**
+
+- **Informal Dates:** Various formats like “روز X ماه سال” and “اول/دوم/… ماه سال”.
+- **Formal Dates:** Stored in YYYY-MM-DD format.
+
+**Example Dates:**
+
+- بیست و هشتم اسفند هزار و چهار صد و ده, 1410-12-28
+- 1 فروردین 1400, 1400-01-01
+
+**Data Split:**
+
+- **Training Set:** 80% (19272 samples)
+- **Validation Set:** 10% (2409 samples)
+- **Test Set:** 10% (2409 samples)
+
+## Model Architecture
+
+**Architecture Details:**  
+The model is built using an encoder-only architecture, consisting of:
+
+- **Layers:** 4 Encoder layers
+- **Parameters:**
+    - `vocab_size`: 25003
+    - `context_length`: 32
+    - `emb_dim`: 256
+    - `n_heads`: 4
+    - `drop_rate`: 0.1
+
+**Parameter Count:** 14,933,931
+
+```
+Transformer( (embedding): Embedding(25003, 256) (positional_encoding): Embedding(32, 256) (en): TransformerEncoder( (layers): ModuleList( (0-3): 4 x TransformerEncoderLayer( (self_attn): MultiheadAttention( (out_proj): NonDynamicallyQuantizableLinear(in_features=256, out_features=256, bias=False) ) (linear1): Linear(in_features=256, out_features=512, bias=False) (dropout): Dropout(p=0.1, inplace=False) (linear2): Linear(in_features=512, out_features=256, bias=False) (norm1): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (norm2): LayerNorm((256,), eps=1e-05, elementwise_affine=True) (dropout1): Dropout(p=0.1, inplace=False) (dropout2): Dropout(p=0.1, inplace=False) ) ) ) (fc_train): Linear(in_features=256, out_features=25003, bias=True) )
+```
+
+**Tokenizer:**  
+The model uses a Persian tokenizer named “بلبل زبان” available on Hugging Face, with a vocabulary size of 25,000 tokens.
+
+## Training
+
+**Training Process:**
+
+- **Batch Size:** 2048
+- **Epochs:** 60
+- **Learning Rate:** 0.00005
+- **Optimizer:** AdamW
+- **Weight Decay:** 0.2
+- **Masking Technique:** The formal part of the date is masked to facilitate learning.
+
+**Performance Metrics:**
+
+- **Training Loss:** Reduced from 10.3 to 0.005 over 60 epochs.
+- **Validation Loss:** Reduced from 10.1 to 0.010.
+- **Test Accuracy:** 66% (exact match required).
+- **Perplexity:** 1.01
+
+## Inference
+
+**Inference Code:**  
+The model can be loaded along with the tokenizer using the provided `Inference.ipynb` file. Three functions are implemented:
+
+1. **Convert Token IDs to Text**
+```python
+def text_to_token_ids(text, tokenizer):
+
+	encoded = tokenizer.encode(text)
+	
+	encoded_tensor = torch.tensor(encoded).unsqueeze(0) # add batch dimension
+
+return encoded_tensor
+```
+
+2. **Convert Text to Token IDs**
+```python
+def token_ids_to_text(token_ids, tokenizer):
+
+	flat = token_ids.squeeze(0) # remove batch dimension
+
+return tokenizer.decode(flat.tolist())
+```
+
+3. **`predict_masked(input)`**: Takes an input to predict the masked date.
+```python
+def predict_masked(model,tokenizer,input,deivce):
+
+	model.eval()
+	
+	inputs_masked = input + " " + "[MASK][MASK][MASK][MASK]-[MASK][MASK]-[MASK][MASK]"
+	
+	input_ids = tokenizer.encode(inputs_masked)
+	
+	input_ids = torch.tensor(input_ids).to(deivce)
+	
+	with torch.no_grad():
+	
+	logits = model(input_ids.unsqueeze(0))
+	
+	logits = logits.flatten(0, 1)
+	
+	probs = torch.argmax(logits,dim=-1,keepdim=True)
+	
+	token_ids = probs.squeeze(1)
+	
+	answer_ids = token_ids[-11:-1]
+
+return token_ids_to_text(answer_ids,tokenizer)
+```
+
+And use:
+```python
+predict_masked(model,tokenizer,"12 آبان 1402","cuda")
+```
+Output: 
+```
+'1402-08-12'
+```
+## Limitations
+
+- The model currently only supports Persian dates for the year 1400-1410, with potential for expansion.
+- Performance may vary with dates outside the training dataset.
+
+## Intended Use
+
+This model is intended for applications requiring date recognition and generation in Persian, such as natural language processing tasks, chatbots, or educational tools.
+
+## Acknowledgements
+
+- Special thanks to the developers of the “بلبل زبان” tokenizer and the contributors to the dataset.