Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,200 @@
+# Indic-Bert-NER-Model
+
+A fine-tuned Named Entity Recognition (NER) model based on [AI4Bharat's Indic-BERT](https://huggingface.co/ai4bharat/indic-bert) for extracting medical and regulatory entities from Indian language documents.
+
+## Model Details
+
+### Overview
+This model is fine-tuned for NER tasks on medical and regulatory documents, specifically for identifying entities in adverse event reports and regulatory submissions. It extends the multilingual Indic-BERT base model with specialized training on pharmaceutical and medical regulatory terminology.
+
+### Model Architecture
+- **Base Model**: [ai4bharat/indic-bert](https://huggingface.co/ai4bharat/indic-bert)
+- **Task**: Token Classification (Named Entity Recognition)
+- **Languages Supported**: Indian languages (Hindi, Tamil, Telugu, Kannada, Malayalam, Bengali, and others)
+- **Framework**: PyTorch / Transformers
+
+### Model Specifications
+- **Model Type**: BERT for token classification
+- **Tokenizer**: SentencePiece
+- **Max Sequence Length**: 512 tokens
+- **Hidden Size**: 768
+- **Number of Attention Heads**: 12
+- **Number of Layers**: 12
+
+## Training Data
+
+The model was trained on the **Indic-Bert-NER-BIO-Dataset**, which includes:
+- Annotated medical and pharmaceutical regulatory documents
+- Multiple data sources: CTRI, FAERS, JSL datasets
+- Phase 2 augmented and merged datasets for improved robustness
+- BIO (Begin-Inside-Outside) tagged entities
+
+For detailed dataset information, see: [Indic-Bert-NER-BIO-Dataset](https://huggingface.co/datasets/redpanda/Indic-Bert-NER-BIO-Dataset)
+
+## Supported Entity Tags
+
+The model recognizes the following entity categories:
+- **Medical Entities**: Drug names, diseases, medical conditions
+- **Regulatory Entities**: Dosages, routes of administration, adverse events
+- **Document Entities**: Document types, regulatory references
+
+Complete entity taxonomy available in the dataset repository.
+
+## Usage
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Basic Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+from transformers import pipeline
+
+# Load model and tokenizer
+model_name = "redpanda/Indic-Bert-NER-Model"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForTokenClassification.from_pretrained(model_name)
+
+# Create NER pipeline
+ner_pipeline = pipeline(
+    "token-classification",
+    model=model,
+    tokenizer=tokenizer,
+    aggregation_strategy="simple"
+)
+
+# Example text
+text = "This drug is made from paracetamol and is used for headache treatment."
+
+# Perform NER
+results = ner_pipeline(text)
+print(results)
+```
+
+### Advanced Usage with Custom Labels
+
+```python
+from transformers import AutoTokenizer, AutoModelForTokenClassification
+import torch
+
+model_name = "redpanda/Indic-Bert-NER-Model"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForTokenClassification.from_pretrained(model_name)
+
+text = "The drug dosage is 500 milligrams daily."
+inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
+
+# Get predictions
+outputs = model(**inputs)
+predictions = torch.argmax(outputs.logits, dim=2)
+
+# Map predictions to labels
+id2label = model.config.id2label
+tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
+
+for token, pred in zip(tokens, predictions[0].numpy()):
+    print(f"{token}: {id2label[pred]}")
+```
+
+### Batch Processing
+
+```python
+from transformers import pipeline
+
+ner = pipeline(
+    "token-classification",
+    model="redpanda/Indic-Bert-NER-Model",
+    aggregation_strategy="simple"
+)
+
+texts = [
+    "Paracetamol is commonly used to treat headaches and fever.",
+    "Take Ibuprofen 400 milligrams tablet for pain relief."
+]
+
+results = [ner(text) for text in texts]
+for text, entities in zip(texts, results):
+    print(f"Text: {text}")
+    print(f"Entities: {entities}\n")
+```
+
+## Model Card
+
+### Model Use
+**Intended Use**: Named Entity Recognition for medical and regulatory documents in Indian languages.
+
+**Primary Users**: 
+- Healthcare professionals
+- Regulatory compliance teams
+- Medical document processors
+- Adverse event monitoring systems
+
+### Limitations
+- Model trained primarily on English-transliterated Indian languages and Hindi
+- Performance may vary on regional language variations
+- Best performance on well-formatted documents
+- Trained on specific pharmaceutical and regulatory domain
+
+### Ethical Considerations
+- Use only for legitimate regulatory and medical purposes
+- Ensure data privacy compliance when processing sensitive health information
+- Do not use for automated decision-making in clinical settings without human review
+- Respect patient confidentiality and HIPAA/DPDP compliance
+
+## License
+
+This model is released under the **MIT License**. 
+
+```
+MIT License
+
+Copyright (c) 2026 CDSCO & ARIP Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+```
+
+## Citation
+
+If you use this model in your research or application, please cite:
+
+```bibtex
+@model{indic_bert_ner_2026,
+  title = {Indic-Bert-NER-Model},
+  author = {CDSCO & ARIP Contributors},
+  year = {2026},
+  url = {https://huggingface.co/redpanda/Indic-Bert-NER-Model},
+  note = {Fine-tuned from AI4Bharat's Indic-BERT}
+}
+```
+
+## Related Resources
+
+- **Base Model**: [AI4Bharat Indic-BERT](https://huggingface.co/ai4bharat/indic-bert)
+- **Dataset**: [Indic-Bert-NER-BIO-Dataset](https://huggingface.co/datasets/redpanda/Indic-Bert-NER-BIO-Dataset)
+- **Training Code**: [ARIP Repository](https://github.com/your-org/arip-platform)
+- **Documentation**: [ARIP Platform Docs](https://github.com/your-org/arip-platform/tree/main/docs)
+
+## Support & Feedback
+
+For issues, questions, or feedback:
+- Open an issue on [GitHub](https://github.com/your-org/arip-platform/issues)
+- Contact: CDSCO & ARIP Team
+
+## Changelog
+
+### Version 1.0 (April 2026)
+- Initial release
+- Fine-tuned on Phase 2 augmented dataset
+- Support for Indian languages via Indic-BERT base

config.json

@@ -0,0 +1,150 @@
+{
+  "architectures": [
+    "AlbertForTokenClassification"
+  ],
+  "attention_probs_dropout_prob": 0,
+  "bos_token_id": 2,
+  "classifier_dropout_prob": 0.1,
+  "down_scale_factor": 1,
+  "dtype": "float32",
+  "embedding_size": 128,
+  "eos_token_id": 3,
+  "gap_size": 0,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "B-AADHAAR",
+    "1": "B-ABHA",
+    "2": "B-ADDRESS_FULL",
+    "3": "B-ADVERSE_EVENT",
+    "4": "B-AGE",
+    "5": "B-CTRI_NUMBER",
+    "6": "B-DATE_DOB",
+    "7": "B-DATE_GENERIC",
+    "8": "B-DIAGNOSIS",
+    "9": "B-DRUG_DOSE",
+    "10": "B-DRUG_NAME",
+    "11": "B-DRUG_ROUTE",
+    "12": "B-EC_REG_NUMBER",
+    "13": "B-EMAIL",
+    "14": "B-LICENSE_NUMBER",
+    "15": "B-LOCATION_CITY",
+    "16": "B-LOCATION_PINCODE",
+    "17": "B-LOCATION_STATE",
+    "18": "B-MRN",
+    "19": "B-ORG_CRO",
+    "20": "B-ORG_EC",
+    "21": "B-ORG_HOSPITAL",
+    "22": "B-ORG_SPONSOR",
+    "23": "B-OUTCOME",
+    "24": "B-PAN",
+    "25": "B-PERSON_GENERIC",
+    "26": "B-PERSON_INVESTIGATOR",
+    "27": "B-PERSON_PATIENT",
+    "28": "B-PHONE_IN",
+    "29": "B-PROTOCOL_NUMBER",
+    "30": "B-SEVERITY",
+    "31": "B-SITE_CODE",
+    "32": "B-VITAL_SIGN",
+    "33": "I-AADHAAR",
+    "34": "I-ABHA",
+    "35": "I-ADDRESS_FULL",
+    "36": "I-ADVERSE_EVENT",
+    "37": "I-AGE",
+    "38": "I-DIAGNOSIS",
+    "39": "I-DRUG_DOSE",
+    "40": "I-DRUG_NAME",
+    "41": "I-EC_REG_NUMBER",
+    "42": "I-LOCATION_STATE",
+    "43": "I-ORG_CRO",
+    "44": "I-ORG_EC",
+    "45": "I-ORG_HOSPITAL",
+    "46": "I-ORG_SPONSOR",
+    "47": "I-OUTCOME",
+    "48": "I-PAN",
+    "49": "I-PERSON_GENERIC",
+    "50": "I-PERSON_INVESTIGATOR",
+    "51": "I-PERSON_PATIENT",
+    "52": "I-PHONE_IN",
+    "53": "I-PROTOCOL_NUMBER",
+    "54": "I-SEVERITY",
+    "55": "I-VITAL_SIGN",
+    "56": "O"
+  },
+  "initializer_range": 0.02,
+  "inner_group_num": 1,
+  "intermediate_size": 3072,
+  "label2id": {
+    "B-AADHAAR": 0,
+    "B-ABHA": 1,
+    "B-ADDRESS_FULL": 2,
+    "B-ADVERSE_EVENT": 3,
+    "B-AGE": 4,
+    "B-CTRI_NUMBER": 5,
+    "B-DATE_DOB": 6,
+    "B-DATE_GENERIC": 7,
+    "B-DIAGNOSIS": 8,
+    "B-DRUG_DOSE": 9,
+    "B-DRUG_NAME": 10,
+    "B-DRUG_ROUTE": 11,
+    "B-EC_REG_NUMBER": 12,
+    "B-EMAIL": 13,
+    "B-LICENSE_NUMBER": 14,
+    "B-LOCATION_CITY": 15,
+    "B-LOCATION_PINCODE": 16,
+    "B-LOCATION_STATE": 17,
+    "B-MRN": 18,
+    "B-ORG_CRO": 19,
+    "B-ORG_EC": 20,
+    "B-ORG_HOSPITAL": 21,
+    "B-ORG_SPONSOR": 22,
+    "B-OUTCOME": 23,
+    "B-PAN": 24,
+    "B-PERSON_GENERIC": 25,
+    "B-PERSON_INVESTIGATOR": 26,
+    "B-PERSON_PATIENT": 27,
+    "B-PHONE_IN": 28,
+    "B-PROTOCOL_NUMBER": 29,
+    "B-SEVERITY": 30,
+    "B-SITE_CODE": 31,
+    "B-VITAL_SIGN": 32,
+    "I-AADHAAR": 33,
+    "I-ABHA": 34,
+    "I-ADDRESS_FULL": 35,
+    "I-ADVERSE_EVENT": 36,
+    "I-AGE": 37,
+    "I-DIAGNOSIS": 38,
+    "I-DRUG_DOSE": 39,
+    "I-DRUG_NAME": 40,
+    "I-EC_REG_NUMBER": 41,
+    "I-LOCATION_STATE": 42,
+    "I-ORG_CRO": 43,
+    "I-ORG_EC": 44,
+    "I-ORG_HOSPITAL": 45,
+    "I-ORG_SPONSOR": 46,
+    "I-OUTCOME": 47,
+    "I-PAN": 48,
+    "I-PERSON_GENERIC": 49,
+    "I-PERSON_INVESTIGATOR": 50,
+    "I-PERSON_PATIENT": 51,
+    "I-PHONE_IN": 52,
+    "I-PROTOCOL_NUMBER": 53,
+    "I-SEVERITY": 54,
+    "I-VITAL_SIGN": 55,
+    "O": 56
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "albert",
+  "net_structure_type": 0,
+  "num_attention_heads": 12,
+  "num_hidden_groups": 1,
+  "num_hidden_layers": 12,
+  "num_memory_blocks": 0,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 2,
+  "vocab_size": 200000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec46315a23d419169d0e1d08ab7ce7c87b5926d22c459dadaa4938630fbc6dee
+size 131590588

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "[SEP]",
+  "unk_token": "<unk>"
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3a1173c2b6e144a02c001e289a05b5dbefddf247c50d4dcf42633158b2968fcb
+size 5646064

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a5325addfc3b3851d746035d0fb75e3efd3d53e000a05e0ca79cc1df5cb954c
+size 15285686

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": {},
+  "keep_accents": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "remove_space": true,
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "AlbertTokenizer",
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:830db2bf011be7a2281886b3573b9ffe271985c931a7ee03176a7e0f90c360c9
+size 5432