scripts/INDICBERT_DOWNLOAD_GUIDE.md

IndicBERT Download Guide - Alternative Methods

Since you're having issues with transformers library, here are 3 alternative methods to download IndicBERT:

---

🚀 Method 1: Using huggingface_hub (RECOMMENDED - Lightest)

This is the SIMPLEST and LIGHTEST method - no transformers needed!

Step 1: Install huggingface_hub

pip install huggingface_hub

Step 2: Run the simple download script

# Option A: Pass token as argument
python scripts/download_indicbert_simple.py your_token_here

# Option B: Set environment variable first
set HUGGINGFACE_TOKEN=your_token_here
python scripts/download_indicbert_simple.py

That's it! This will download the model files to your cache directory.

---

🔧 Method 2: Using Git LFS

If you have Git and Git LFS installed:

# Install git-lfs (if not installed)
# Windows: Download from https://git-lfs.github.com/
# Then:
git lfs install
git clone https://your_token@huggingface.co/ai4bharat/indic-bert ./models/indic-bert
cd ./models/indic-bert
git lfs pull

---

🌐 Method 3: Direct API Download

Uses Python requests library to download files directly:

python scripts/download_indicbert_alternative.py your_token_here

This script tries all 3 methods automatically.

---

📋 What You Need to Provide

1. Your HuggingFace Token

   • Get it from: https://huggingface.co/settings/tokens
   • Should start with hf_
   • Make sure it has "Read" permissions

2. Model Access

   • Request access at: https://huggingface.co/ai4bharat/indic-bert
   • Click "Agree and access repository"
   • Wait for approval (usually instant)

---

✅ Quick Test

Once you have your token, test it:

# Install the lightweight library
pip install huggingface_hub

# Test download
python scripts/download_indicbert_simple.py your_token_here

---

🐛 Troubleshooting

Error: "401 Unauthorized"

• Fix: Check your token is correct
• Fix: Make sure you've requested access to the model

Error: "Gated repository"

• Fix: Request access at https://huggingface.co/ai4bharat/indic-bert
• Fix: Wait for approval, then try again

Error: "huggingface_hub not found"

• Fix: pip install huggingface_hub

Error: "Model not found"

• Fix: Check model ID: ai4bharat/indic-bert
• Fix: Verify it exists at HuggingFace

---

📍 Model Location After Download

After successful download, the model will be cached at:

• Windows: C:\Users\<username>\.cache\huggingface\hub\models--ai4bharat--indic-bert
• Linux/Mac: ~/.cache/huggingface/hub/models--ai4bharat--indic-bert

You can use it later with:

from transformers import AutoModel
model = AutoModel.from_pretrained("ai4bharat/indic-bert", token="your_token")

---

🎯 Recommended Approach

Use Method 1 (huggingface_hub) - it's the simplest and doesn't require transformers!

pip install huggingface_hub
python scripts/download_indicbert_simple.py your_token_here