MWirelabs
/

assamese-roberta

@@ -1,199 +1,207 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
 ## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 ### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
 ### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+language:
+- as
+license: cc-by-4.0
+tags:
+- assamese
+- roberta
+- masked-lm
+- fill-mask
+datasets:
+- MWirelabs/assamese-monolingual-corpus
+metrics:
+- perplexity
+model-index:
+- name: AssameseRoBERTa
+  results:
+  - task:
+      type: fill-mask
+      name: Masked Language Modeling
+    metrics:
+    - name: Perplexity (Training Domain)
+      type: perplexity
+      value: 1.5738
+    - name: Perplexity (Unseen Text)
+      type: perplexity
+      value: 5.9281
 ---
+# AssameseRoBERTa
+## Model Description
+AssameseRoBERTa is a RoBERTa-based language model trained from scratch on Assamese monolingual text. The model is designed to provide robust language understanding capabilities for the Assamese language, which is spoken by over 15 million people primarily in the Indian state of Assam.
+This model was developed by [MWire Labs](https://mwirelabs.com), an AI research organization focused on building language technologies for Northeast Indian languages.
 ## Model Details
+- **Model Type:** RoBERTa (Robustly Optimized BERT Pretraining Approach)
+- **Language:** Assamese (as)
+- **Training Data:** 1.6M Assamese sentences from diverse sources
+- **Parameters:** ~125M
+- **Training Epochs:** 10
+- **Training Duration:** 8 hours on A40 GPU
+- **Vocabulary Size:** ~50,000 tokens
+- **Max Sequence Length:** 512 tokens
+## Performance
+### Perplexity Scores
+The model achieves strong performance on both in-domain and out-of-domain evaluation:
+| Model | Training Domain PPL | Unseen Text PPL |
+|-------|---------------------|-----------------|
+| **AssameseRoBERTa (Ours)** | **1.5738** | **5.9281** |
+| mBERT | 29.8206 | 9.9891 |
+| MuRIL | 27.3264 | 14.2509 |
+| Assamese-BERT | 12.1166 | 22.6595 |
+| IndicBERT | - | 283.7512 |
+The model significantly outperforms existing multilingual models on Assamese text, demonstrating the value of language-specific pretraining.
+## Intended Use
 ### Direct Use
+This model is intended for:
+- Masked language modeling tasks
+- Feature extraction for downstream Assamese NLP tasks
+- Fine-tuning on Assamese-specific tasks such as:
+  - Text classification
+  - Named Entity Recognition (NER)
+  - Sentiment analysis
+  - Question answering
+  - Token classification
 ### Out-of-Scope Use
+This model should not be used for:
+- Generating factual information without verification
+- Making decisions that affect individuals' rights or well-being without human oversight
+- Any application requiring real-time critical decision making
+## Training Data
+The model was trained on the [MWirelabs/assamese-monolingual-corpus](https://huggingface.co/datasets/MWirelabs/assamese-monolingual-corpus) dataset, which contains approximately 1.6 million Assamese sentences from diverse sources including:
+- News articles
+- Literature
+- Web crawl data
+- Government documents
+- Social media content
+The diverse nature of the training data helps the model generalize across different domains and text styles.
+## Training Procedure
+### Preprocessing
+- Text normalization for Assamese script (Bengali script)
+- Tokenization using SentencePiece
+- Vocabulary built specifically for Assamese
+### Training Hyperparameters
+- **Architecture:** RoBERTa-base
+- **Optimizer:** AdamW
+- **Learning Rate:** Peak LR with warmup and linear decay
+- **Batch Size:** Optimized for A40 GPU
+- **Training Epochs:** 10
+- **Hardware:** NVIDIA A40 (48GB)
+- **Precision:** Mixed precision (BF16)
+- **Training Time:** ~8 hours
 ## Evaluation
+The model was evaluated on both in-domain and out-of-domain Assamese text using perplexity as the primary metric. The significantly lower perplexity compared to multilingual baselines demonstrates strong language modeling capabilities.
+## Usage
+### Using Transformers
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("MWirelabs/assamese-roberta")
+model = AutoModelForMaskedLM.from_pretrained("MWirelabs/assamese-roberta")
+# Example: Fill mask
+text = "অসম হৈছে [MASK] এখন সুন্দৰ ৰাজ্য।"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+# Get predictions
+masked_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
+predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
+predicted_token = tokenizer.decode(predicted_token_id)
+print(f"Predicted token: {predicted_token}")
+```
+### Feature Extraction
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+tokenizer = AutoTokenizer.from_pretrained("MWirelabs/assamese-roberta")
+model = AutoModel.from_pretrained("MWirelabs/assamese-roberta")
+text = "অসমীয়া ভাষা অতি সুন্দৰ।"
+inputs = tokenizer(text, return_tensors="pt")
+with torch.no_grad():
+    outputs = model(**inputs)
+    embeddings = outputs.last_hidden_state
+print(f"Embeddings shape: {embeddings.shape}")
+```
+## Limitations
+- The model is trained exclusively on Assamese text and does not perform well on other languages
+- Performance may vary on specialized domains not well-represented in the training data
+- The model inherits biases present in the training data
+- Code-mixed text (Assamese-English) may not be handled optimally
+## Ethical Considerations
+- This model may reflect biases present in the training corpus
+- Users should evaluate the model's outputs in their specific context before deployment
+- The model should not be used for generating harmful or misleading content
+- Consider fairness implications when deploying in real-world applications
+## Citation
+If you use this model in your research, please cite:
+```bibtex
+@misc{assamese-roberta-2024,
+  author = {MWire Labs},
+  title = {AssameseRoBERTa: A RoBERTa Model for Assamese Language},
+  year = {2024},
+  publisher = {HuggingFace},
+  howpublished = {\url{https://huggingface.co/MWirelabs/assamese-roberta}}
+}
+```
+## Contact
+For questions or feedback, please contact:
+- Website: https://mwirelabs.com
+- Email: connect@mwirelabs.com
+## License
+This model is released under the **Creative Commons Attribution 4.0 International License (CC-BY-4.0)**.
+You are free to:
+- **Share** — copy and redistribute the material in any medium or format
+- **Adapt** — remix, transform, and build upon the material for any purpose, even commercially
+Under the following terms:
+- **Attribution** — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
+See the full license at: https://creativecommons.org/licenses/by/4.0/