Update README.md

README.md

@@ -3,6 +3,9 @@ tags:
 - tangkhul
 - corpus
 - BERT
+- fill-mask
+- text-generation
+- low-resource-language
 license: apache-2.0
 base_model:
 - google-bert/bert-base-uncased
@@ -10,25 +13,26 @@ base_model:
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
+This repository contains TangkhulBERT, the first publicly available foundational language model for the Tangkhul language, a low-resource Tibeto-Burman language. The model was trained from scratch using a Masked Language Modeling (MLM) objective.
 
 
 
 ## Model Details
-
+Model Description
+TangkhulBERT is a transformer-based model with a BERT-base architecture. It was developed to provide a crucial NLP resource for the Tangkhul language community and to serve as a starting point for various downstream tasks.
 ### Model Description
 
 <!-- Provide a longer summary of what this model is. -->
 
-This repository contains TangkhulBERT, the first publicly available foundational language model for the Tangkhul language, a low-resource Tibeto-Burman language. The model was trained from scratch using a Masked Language Modeling (MLM) objective.
+
 
 - **Developed by:** Vinos shimray
 - **Funded by [optional]:** [More Information Needed]
 - **Shared by [optional]:** [More Information Needed]
 - **Model type:** BERT Base
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Language(s) (NLP):** Tangkhul
+- **License:** apache-2.0
+- **Finetuned from model [optional]:** This model was trained from scratch and not fine-tuned from any other model.
 
 ### Model Sources [optional]
 
@@ -44,27 +48,47 @@ This repository contains TangkhulBERT, the first publicly available foundational
 
 ### Direct Use
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+The model is intended for direct use in Masked Language Modeling tasks.
+
+from transformers import pipeline
+
+fill_mask = pipeline(
+    "fill-mask",
+    model="vinshim/TangkhulBERT",
+    tokenizer="VinosShimray/TangkhulBERT"
+)
+
+# Test with a Tangkhul sentence
+result = fill_mask(" [MASK].")
+
+# Print the top predictions
+for prediction in result:
+    print(prediction)
+
 
-[More Information Needed]
 
 ### Downstream Use [optional]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+This model is designed to be a foundational, pre-trained model for fine-tuning on specific downstream tasks such as:
 
-[More Information Needed]
+Text Classification (e.g., sentiment analysis, topic categorization)
 
-### Out-of-Scope Use
+Named Entity Recognition (NER)
+
+Question Answering
+
+Machine Translation (as an encoder)
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-[More Information Needed]
+
+### Out-of-Scope Use
+
+This model is not intended for generating long-form, coherent text. Due to the limited size of the training corpus, it should not be used in safety-critical applications or for tasks requiring deep, nuanced world knowledge. The model only understands Tangkhul and will not perform well on other languages.
 
 ## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+The primary limitation is the size of the pre-training corpus (4 MB). While significant for a low-resource language, this is small compared to models for high-resource languages. The model will reflect any biases present in the source text data. Its knowledge is confined to the domains covered in the training corpus and may not generalize well to other contexts.
 
-[More Information Needed]
 
 ### Recommendations
 
@@ -74,33 +98,55 @@ Users (both direct and downstream) should be made aware of the risks, biases and
 
 ## How to Get Started with the Model
 
-Use the code below to get started with the model.
+Use the code below to get started with the model for Masked Language Modeling.
+from transformers import pipeline, AutoTokenizer, AutoModelForMaskedLM
 
-[More Information Needed]
+# Replace with your Hugging Face username and repo name
+repo_id = "vinshim/TangkhulBERT"
+
+# Load tokenizer and model
+tokenizer = AutoTokenizer.from_pretrained(repo_id)
+model = AutoModelForMaskedLM.from_pretrained(repo_id)
 
+# Create the pipeline
+fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
+
+# Use the model
+result = fill_mask("Kazing eina ngalei [MASK].")
+print(result)
 ## Training Details
 
 ### Training Data
 
 <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was pre-trained on a 4 MB plain-text corpus of the Tangkhul language, collected from various digital sources. This data is not available for download but can be described as general-purpose text.
 
-[More Information Needed]
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
 #### Preprocessing [optional]
+The text was preprocessed by:
 
+1.Converting all text to lowercase.
+
+2. Ensuring a sentence-per-line format.
+
+3. Programmatically adding a full stop (.) to every line that lacked sentence-ending punctuation.
 [More Information Needed]
 
 
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Training regime:** fp16 mixed precision
+Epochs: 500
+Batch Size: 128
+Optimizer: AdamW with default settings
+Learning Rate: 5e-5 <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
 #### Speeds, Sizes, Times [optional]
-
+The pre-training was conducted over approximately 3 hours on a single NVIDIA A100 GPU.
 <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
 [More Information Needed]
@@ -124,14 +170,12 @@ Use the code below to get started with the model.
 [More Information Needed]
 
 #### Metrics
-
+The primary evaluation metric during pre-training was Training Loss, which is a measure of the model's perplexity on the Masked Language Modeling task.
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
 
 ### Results
-
-[More Information Needed]
+The model achieved a final pre-training loss of 2.9969 after 22,000 training steps.
 
 #### Summary