Sync: correct languages (en/ne/es/fr/ar/hi/te/zh/ja/ko/pt), updated README, full source code

README.md

@@ -2,26 +2,29 @@
 language:
   - en
   - ne
-  - hi
-  - ar
-  - fr
   - es
-  - sw
-  - id
-  - pt
+  - fr
+  - ar
+  - hi
+  - te
   - zh
-  - bn
+  - ja
+  - ko
+  - pt
 license: llama3
 library_name: transformers
+base_model: meta-llama/Llama-3.1-8B-Instruct
 tags:
-  - disaster-response
-  - emergency-management
+  - disaster-management
+  - emergency-response
   - humanitarian-ai
   - multilingual
   - fine-tuned
   - qlora
+  - lora
+  - peft
   - llama3
-base_model: meta-llama/Llama-3.1-8B-Instruct
+pipeline_tag: text-generation
 model-index:
   - name: WorldDisasterLM-8B
     results: []
@@ -29,24 +32,25 @@ model-index:
 
 # WorldDisasterLM-8B
 
-> **Open-source AI foundation model for global disaster intelligence, emergency response, and humanitarian aid — supporting 11 languages including Nepali.**
+> **Open-source AI foundation model for global disaster intelligence, emergency response, and humanitarian aid — supporting 11 languages including Nepali (नेपाली).**
 
 [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://python.org)
 [![License](https://img.shields.io/badge/license-Llama3-green.svg)](https://ai.meta.com/llama/license/)
-[![Space](https://img.shields.io/badge/🤗%20Space-Live%20Demo-yellow)](https://huggingface.co/spaces/drdeveloper88/WorldDisasterLM-8B)
+[![HuggingFace](https://img.shields.io/badge/🤗-WorldDisasterLM--8B-yellow)](https://huggingface.co)
+[![Tests](https://img.shields.io/badge/tests-9%20passed-brightgreen.svg)]()
 
 ---
 
-## Model Description
+## What is WorldDisasterLM-8B?
 
 **WorldDisasterLM-8B** is a production-grade, domain-specialized large language model fine-tuned on top of `meta-llama/Llama-3.1-8B-Instruct` using **QLoRA** (4-bit NF4 quantization, LoRA r=16). It is purpose-built to assist:
 
-- **Emergency responders** — real-time disaster action guidance
-- **Humanitarian aid workers** — resource allocation and triage support  
-- **Government agencies** — risk assessment and crisis intelligence
-- **Global communities** — multilingual disaster preparedness in 11 languages
+- 🌍 **Emergency responders** — real-time disaster action guidance
+- 🏥 **Humanitarian aid workers** — resource allocation and triage support
+- 🏛️ **Government agencies** — risk assessment and crisis intelligence
+- 🌐 **Global communities** — multilingual disaster preparedness in 11 languages
 
-Training data is collected live from six free public APIs: ReliefWeb, USGS Earthquake, GDACS, NOAA Weather, OpenFEMA, and WHO — with automated QA amplification generating 8 instruction variants per disaster record.
+Training data is collected **live** from six free public APIs: ReliefWeb, USGS Earthquake, GDACS, NOAA Weather, OpenFEMA, and WHO — with automated QA amplification generating 8 instruction variants per disaster record.
 
 ---
 
@@ -56,192 +60,288 @@ Training data is collected live from six free public APIs: ReliefWeb, USGS Earth
 |---|---|
 | **Base model** | `meta-llama/Llama-3.1-8B-Instruct` |
 | **Fine-tuning** | QLoRA — 4-bit NF4, LoRA r=16, all attn+MLP projectors |
-| **Languages** | 11: English, Nepali, Hindi, Arabic, French, Spanish, Swahili, Indonesian, Portuguese, Chinese, Bengali |
-| **API** | FastAPI REST with `/v1/chat`, `/v1/risk/score`, `/v1/incidents/classify` |
-| **Training data** | ReliefWeb, USGS, GDACS, NOAA, OpenFEMA, WHO |
-| **Special feature** | NDRRMA citations for Nepali disaster queries |
+| **Languages** | 11: English, Nepali (नेपाली), Spanish, French, Arabic, Hindi, Telugu, Chinese, Japanese, Korean, Portuguese |
+| **Data sources** | ReliefWeb, USGS, GDACS, NOAA, OpenFEMA, WHO |
+| **Dataset size** | 88+ live records → 711+ instruction samples per run |
+| **API** | FastAPI REST + Server-Sent Events streaming |
+| **Frontend** | React 18 + Vite disaster analytics dashboard |
+| **Export** | GGUF (llama.cpp) + ONNX formats |
+| **HuggingFace** | Auto-push via `scripts/push_to_hub.py` |
 
 ---
 
-## Live Demo
+## Supported Languages
 
-Try the model without any setup at the [WorldDisasterLM-8B Gradio Space](https://huggingface.co/spaces/drdeveloper88/WorldDisasterLM-8B) — available in all 11 languages.
+| Language | Code | Script |
+|---|---|---|
+| English | `en` | Latin |
+| **Nepali** | `ne` | **Devanagari (नेपाली)** |
+| Spanish | `es` | Latin |
+| French | `fr` | Latin |
+| Arabic | `ar` | Arabic |
+| Hindi | `hi` | Devanagari |
+| Telugu | `te` | Telugu |
+| Chinese | `zh` | Hanzi |
+| Japanese | `ja` | Kanji/Hiragana |
+| Korean | `ko` | Hangul |
+| Portuguese | `pt` | Latin |
 
 ---
 
-## Repository Structure
+## Quick Start
 
-```
-WorldDisasterLM-8B/
-├── backend/              # FastAPI REST API (production server)
-│   └── app/
-│       ├── main.py       # App entry point, /v1/* routes
-│       ├── routers/      # chat, risk, incidents endpoints
-│       └── services/     # inference_service.py
-├── worlddisasterlm/      # Core model package
-│   ├── config.py         # SUPPORTED_LANGUAGES, model config
-│   ├── model.py          # QLoRA model wrapper
-│   └── trainer.py        # Training pipeline
-├── hf_space/             # Gradio Space app (self-contained)
-│   ├── app.py
-│   └── requirements.txt
-├── scripts/              # Training, eval, dataset scripts
-│   ├── train_production.py
-│   └── generate_dataset.py
-├── tests/                # 9 passing unit/integration tests
-├── train.py              # Main training entry point
-├── evaluate.py           # Evaluation harness
-├── inference.py          # Direct inference script
-├── dataset_builder.py    # Data pipeline
-├── Dockerfile            # Production container
-├── docker-compose.yml    # Multi-service setup
-└── requirements.txt      # Python dependencies
-```
+### 1. Clone and install
 
----
+```bash
+git clone <your-repo-url> worlddisasterllm
+cd worlddisasterllm
+python -m venv .venv
 
-## Quick Start
+# Windows
+.venv\Scripts\activate
+# macOS/Linux
+source .venv/bin/activate
+
+pip install -r requirements.txt
+```
 
-### 1. Clone and Install
+> **Windows note:** If your path has spaces, always set:
+> ```powershell
+> $env:PYTHONPATH = "C:\path\to\worlddisasterllm"
+> ```
+
+### 2. Configure environment
 
 ```bash
-git clone https://huggingface.co/drdeveloper88/WorldDisasterLM-8B
-cd WorldDisasterLM-8B
-pip install -r requirements.txt
+cp .env.example .env
+# Add HF_TOKEN for HuggingFace publishing (optional)
 ```
 
-### 2. Run the FastAPI Server
+### 3. Run the API backend
 
 ```bash
-uvicorn backend.app.main:app --host 0.0.0.0 --port 8000
+uvicorn backend.app.main:app --reload --port 8000
 ```
 
-### 3. Chat API
+API available at: `http://localhost:8000`  
+Interactive docs: `http://localhost:8000/docs`
+
+### 4. Run the React dashboard
 
 ```bash
-curl -X POST http://localhost:8000/v1/chat \
-  -H "Content-Type: application/json" \
-  -d '{"message": "Earthquake safety tips", "language": "en", "region": "Nepal"}'
+cd frontend
+npm install
+npm run dev
 ```
 
-### 4. Risk Scoring
+Dashboard at: `http://localhost:5173`
+
+### 5. Run the Gradio demo
 
 ```bash
-curl -X POST http://localhost:8000/v1/risk/score \
-  -H "Content-Type: application/json" \
-  -d '{"region": "Kathmandu", "hazard": "earthquake", "vulnerability": 0.8, "exposure": 0.9}'
+python app.py
 ```
 
-### 5. Docker (Recommended)
+Demo at: `http://localhost:7860`
+
+### 6. Run CLI inference
 
 ```bash
-docker-compose up --build
+# English
+python inference.py --prompt "What to do during a flood?" --language English --region Nepal
+
+# Nepali
+python inference.py --prompt "भूकम्पको बेला के गर्ने?" --language Nepali --region Nepal
 ```
 
 ---
 
-## Supported Languages
+## Training Pipeline
 
-| Code | Language | Script |
-|------|----------|--------|
-| `en` | English | Latin |
-| `ne` | Nepali | Devanagari |
-| `hi` | Hindi | Devanagari |
-| `ar` | Arabic | Arabic |
-| `fr` | French | Latin |
-| `es` | Spanish | Latin |
-| `sw` | Swahili | Latin |
-| `id` | Indonesian | Latin |
-| `pt` | Portuguese | Latin |
-| `zh` | Chinese | CJK |
-| `bn` | Bengali | Bengali |
+### Step 1 — Collect live data
 
----
+```bash
+python scripts/collect_data.py
+# Collects from: ReliefWeb, USGS, GDACS, NOAA, OpenFEMA, WHO
+# Output: data/processed/instruction_dataset.jsonl
+```
 
-## API Endpoints
+### Step 2 — Fine-tune with QLoRA
 
-| Endpoint | Method | Description |
-|----------|--------|-------------|
-| `/health` | GET | Health check |
-| `/v1/chat` | POST | Disaster Q&A in any language |
-| `/v1/risk/score` | POST | Risk score for region/hazard |
-| `/v1/incidents/classify` | POST | Classify incident type |
+```bash
+python train.py --output checkpoints/worlddisasterlm-8b-qlora
+# Requires GPU (A100/V100 recommended)
+# Uses: 4-bit NF4 quantization + LoRA r=16 + SFTTrainer
+```
 
----
+### Step 3 — Evaluate
 
-## Training
+```bash
+python evaluate.py
+```
 
-Uses QLoRA for efficient fine-tuning on a single GPU:
+### Step 4 — Push to HuggingFace
 
 ```bash
-python train.py \
-  --model_name meta-llama/Llama-3.1-8B-Instruct \
-  --lora_r 16 \
-  --lora_alpha 32 \
-  --bits 4 \
-  --output_dir ./checkpoints
+export HF_TOKEN=hf_xxxx   # or $env:HF_TOKEN on Windows
+python scripts/push_to_hub.py \
+    --adapter checkpoints/worlddisasterlm-8b-qlora \
+    --base-model meta-llama/Llama-3.1-8B-Instruct \
+    --repo-id YourUsername/WorldDisasterLM-8B
 ```
 
-Or use the production training script:
+### Step 5 — Export to GGUF (for llama.cpp / Ollama)
 
 ```bash
-python scripts/train_production.py
+python scripts/export_gguf.py \
+    --model-path checkpoints/worlddisasterlm-8b-qlora \
+    --output-path artifacts/worlddisasterlm-8b.gguf
 ```
 
 ---
 
-## Evaluation
+## API Reference
 
-```bash
-python evaluate.py --model_path ./checkpoints/final
+### Health check
+```
+GET /health
+→ { "status": "ok", "model": "WorldDisasterLM-8B" }
+```
+
+### Disaster chat
+```
+POST /v1/chat
+{
+  "messages": [{"role": "user", "content": "What to do during earthquake?"}],
+  "language": "Nepali",
+  "region": "Nepal"
+}
+→ { "answer": "[WorldDisasterLM-8B | नेपाली | Nepal] ...", "confidence": 0.74, ... }
+```
+
+### Streaming chat
+```
+POST /v1/chat/stream    (Server-Sent Events)
+```
+
+### Risk scoring
+```
+POST /v1/risk/score
+{ "region": "South Asia", "hazard_type": "flood", "vulnerability_index": 0.74, "exposure_index": 0.81 }
+→ { "risk_score": ..., "risk_level": "High", "recommendation": "..." }
+```
+
+### Incident classification
+```
+POST /v1/incidents/classify
+{ "text": "Magnitude 7.1 earthquake near coastal city. Hospitals overloaded." }
+→ { "incident_type": "earthquake", "severity": "critical", ... }
 ```
 
-Metrics: response accuracy, hallucination rate, safety compliance, emergency-response correctness, multilingual performance.
+---
+
+## Repository Structure
+
+```
+worlddisasterlm/
+├── backend/                   FastAPI backend
+│   └── app/
+│       ├── main.py            API entrypoint
+│       ├── services/
+│       │   ├── inference_service.py   Core response generation (multilingual)
+│       │   └── risk_engine.py         Risk scoring engine
+│       ├── guardrails/
+│       │   └── safety.py      Prompt safety filters
+│       └── models/schemas.py  Pydantic v2 request/response models
+├── worlddisasterlm/           Core ML package
+│   ├── config.py              SUPPORTED_LANGUAGES, PipelineConfig
+│   ├── data/
+│   │   ├── collectors/        6 live API data collectors
+│   │   ├── etl.py             Extract-Transform-Load pipeline
+│   │   ├── qa_generator.py    8-variant QA amplification
+│   │   ├── scenario_builder.py  Multilingual disaster scenarios
+│   │   └── processors.py      JSONL dataset builder
+│   ├── training/
+│   │   ├── train_qlora.py     Production QLoRA training (SFTTrainer)
+│   │   └── chat_format.py     Llama 3.1 chat template
+│   ├── evaluation/
+│   │   ├── metrics.py         Task + safety metrics
+│   │   └── multilingual_eval.py  11-language coverage scoring
+│   └── optimization/
+│       ├── export_gguf.py     GGUF export (llama.cpp)
+│       └── export_onnx.py     ONNX export
+├── frontend/                  React 18 + Vite dashboard
+│   └── src/App.jsx            Language selector, chat, analytics, monitoring
+├── scripts/
+│   ├── collect_data.py        Live data collection orchestrator
+│   ├── push_to_hub.py         HuggingFace Hub publisher
+│   ├── export_gguf.py         GGUF CLI wrapper
+│   └── export_onnx.py         ONNX CLI wrapper
+├── tests/
+│   ├── test_api.py            FastAPI integration tests (incl. Nepali)
+│   ├── test_dataset_builder.py  ETL + Nepali language tests
+│   └── test_risk_engine.py    Risk scoring tests
+├── inference.py               CLI inference entrypoint
+├── train.py                   Training entrypoint
+├── evaluate.py                Evaluation entrypoint
+├── app.py                     Gradio demo
+├── dataset_builder.py         Dataset build entrypoint
+├── Dockerfile                 API container
+├── docker-compose.yml         Full stack (API + frontend + MLflow)
+└── pyproject.toml             Project metadata + pytest config
+```
 
 ---
 
-## Intended Use
+## Running Tests
 
-- Government disaster agencies and civil protection bodies
-- NGOs and humanitarian organizations (UN OCHA, Red Cross, etc.)
-- Emergency responders and first responders
-- Disaster risk researchers and policy planners
-- Healthcare organizations in crisis zones
-- Community preparedness programs
+```bash
+# Windows
+$env:PYTHONPATH = "C:\path\to\worlddisasterllm"
+python -m pytest tests/ -v
 
-## Out-of-Scope Use
+# macOS/Linux
+PYTHONPATH=. pytest tests/ -v
+```
 
-- Real-time operational dispatch (use certified emergency systems)
-- Medical diagnosis or clinical decisions
-- Financial or legal advice
-- Any purpose that replaces trained human emergency professionals
+Expected output: **9 passed** — including Nepali language tests.
 
 ---
 
-## Safety & Limitations
+## Safety and Responsible AI
 
-- **Not a replacement** for official emergency command centers
-- Outputs should be **verified** with authoritative real-time sources
-- High-risk outputs are tagged for escalation and human review
-- Data availability and timeliness may vary by region
-- Low-resource languages may have lower response quality
+- Prompt-level guardrails block unsafe instructions
+- Output confidence scoring with human-review flags
+- Emergency disclaimers on all life-safety guidance
+- Hallucination heuristics with citation anchoring
+- Human-in-the-loop escalation for critical incidents
 
 ---
 
-## Citation
+## HuggingFace Publishing
 
-```bibtex
-@misc{worlddisasterlm2025,
-  title={WorldDisasterLM-8B: A Multilingual Foundation Model for Disaster Intelligence},
-  author={drdeveloper88},
-  year={2025},
-  publisher={HuggingFace},
-  url={https://huggingface.co/drdeveloper88/WorldDisasterLM-8B}
-}
-```
+The model is published as **`WorldDisasterLM-8B`** on HuggingFace Hub with:
+- Merged LoRA adapters (production-ready weights)
+- Model card with 11-language tags (`en`, `ne`, `es`, `fr`, `ar`, `hi`, `te`, `zh`, `ja`, `ko`, `pt`)
+- GGUF variant for llama.cpp / Ollama compatibility
 
 ---
 
 ## License
 
-Based on [Meta Llama 3](https://ai.meta.com/llama/license/) — usage governed by the Llama 3 Community License Agreement.
+Base model: [Llama 3 Community License](https://ai.meta.com/llama/license/) (Meta).  
+This fine-tuning code and dataset pipeline: **Apache 2.0**.
+
+---
+
+## Acknowledgements
+
+- [Meta AI — Llama 3.1](https://ai.meta.com/blog/meta-llama-3/)
+- [ReliefWeb API](https://reliefweb.int/help/api)
+- [USGS Earthquake Hazards Program](https://earthquake.usgs.gov/fdsnws/event/1/)
+- [GDACS Global Disaster Alert](https://www.gdacs.org)
+- [NOAA Weather.gov API](https://api.weather.gov)
+- [OpenFEMA API](https://www.fema.gov/about/openfema)
+- [WHO RSS Feeds](https://www.who.int/rss-feeds)
+- [NDRRMA Nepal — राष्ट्रिय विपद् जोखिम न्यूनीकरण तथा व्यवस्थापन प्राधिकरण](https://www.ndrrma.gov.np)
+

scripts/sync_to_hf.py

@@ -0,0 +1,380 @@
+"""
+Fully automated sync of WorldDisasterLM-8B project to HuggingFace model repo.
+Reads the real project README, updates all configs to match, commits and pushes.
+
+Usage:
+    python scripts/sync_to_hf.py
+"""
+import json
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+# ── Paths ─────────────────────────────────────────────────────────────────────
+PROJECT_ROOT = Path(__file__).parent.parent
+REPO_DIR = Path(os.environ.get("HF_REPO_DIR", r"C:\Users\naidu\AppData\Local\Temp\wdlm_model_push"))
+HF_TOKEN  = os.environ.get("HF_TOKEN", "")
+HF_REPO   = "drdeveloper88/WorldDisasterLM-8B"
+
+# ── Correct language list from project README ─────────────────────────────────
+LANGUAGES = ["en", "ne", "es", "fr", "ar", "hi", "te", "zh", "ja", "ko", "pt"]
+LANGUAGE_NAMES = {
+    "en": "English", "ne": "Nepali (नेपाली)", "es": "Spanish",
+    "fr": "French",  "ar": "Arabic",           "hi": "Hindi",
+    "te": "Telugu",  "zh": "Chinese",           "ja": "Japanese",
+    "ko": "Korean",  "pt": "Portuguese",
+}
+
+# ── 1. Build the HF model card README ────────────────────────────────────────
+def build_readme():
+    # Read the actual project README content (strip the first H1 title line)
+    src_readme = (PROJECT_ROOT / "README.md").read_text(encoding="utf-8")
+
+    yaml_header = f"""---
+language:
+{chr(10).join(f"  - {lang}" for lang in LANGUAGES)}
+license: llama3
+library_name: transformers
+base_model: meta-llama/Llama-3.1-8B-Instruct
+tags:
+  - disaster-management
+  - emergency-response
+  - humanitarian-ai
+  - multilingual
+  - fine-tuned
+  - qlora
+  - lora
+  - peft
+  - llama3
+pipeline_tag: text-generation
+model-index:
+  - name: WorldDisasterLM-8B
+    results: []
+---
+
+"""
+    return yaml_header + src_readme
+
+
+# ── 2. Config files ───────────────────────────────────────────────────────────
+def build_configs():
+    SYSTEM_PROMPT = (
+        "You are WorldDisasterLM-8B, an expert AI specialized in global disaster "
+        "management, emergency response, and humanitarian aid. You provide accurate, "
+        "actionable guidance in 11 languages: English, Nepali (नेपाली), Spanish, "
+        "French, Arabic, Hindi, Telugu, Chinese, Japanese, Korean, and Portuguese. "
+        "Always prioritize life safety. Cite authoritative sources (NDRRMA for Nepal, "
+        "WHO, FEMA, USGS, GDACS) when relevant. Never provide false hope or inaccurate "
+        "information in emergency situations."
+    )
+
+    CHAT_TEMPLATE = (
+        "{%- set default_system = \"" + SYSTEM_PROMPT.replace('"', "'") + "\" %}"
+        "{%- if messages[0]['role'] == 'system' %}"
+        "{%- set default_system = messages[0]['content'] %}"
+        "{%- set messages = messages[1:] %}"
+        "{%- endif %}"
+        "{{ bos_token }}"
+        "<|start_header_id|>system<|end_header_id|>\n\n{{ default_system }}<|eot_id|>"
+        "{%- for message in messages %}"
+        "{{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }}"
+        "{%- endfor %}"
+        "{%- if add_generation_prompt %}"
+        "{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}"
+        "{%- endif %}"
+    )
+
+    configs = {
+        "config.json": {
+            "_name_or_path": "drdeveloper88/WorldDisasterLM-8B",
+            "architectures": ["LlamaForCausalLM"],
+            "attention_bias": False,
+            "attention_dropout": 0.0,
+            "bos_token_id": 128000,
+            "eos_token_id": [128001, 128008, 128009],
+            "head_dim": 128,
+            "hidden_act": "silu",
+            "hidden_size": 4096,
+            "initializer_range": 0.02,
+            "intermediate_size": 14336,
+            "max_position_embeddings": 131072,
+            "mlp_bias": False,
+            "model_type": "llama",
+            "num_attention_heads": 32,
+            "num_hidden_layers": 32,
+            "num_key_value_heads": 8,
+            "pretraining_tp": 1,
+            "rms_norm_eps": 1e-05,
+            "rope_interleaved": False,
+            "rope_scaling": {
+                "factor": 8.0,
+                "high_freq_factor": 4.0,
+                "low_freq_factor": 1.0,
+                "original_max_position_embeddings": 8192,
+                "rope_type": "llama3",
+            },
+            "rope_theta": 500000.0,
+            "tie_word_embeddings": False,
+            "torch_dtype": "bfloat16",
+            "transformers_version": "4.43.0",
+            "use_cache": True,
+            "vocab_size": 128256,
+            "quantization_config": {
+                "quant_method": "bitsandbytes",
+                "load_in_4bit": True,
+                "load_in_8bit": False,
+                "bnb_4bit_quant_type": "nf4",
+                "bnb_4bit_compute_dtype": "bfloat16",
+                "bnb_4bit_use_double_quant": True,
+                "bnb_4bit_quant_storage": "uint8",
+                "llm_int8_threshold": 6.0,
+                "llm_int8_skip_modules": None,
+                "llm_int8_enable_fp32_cpu_offload": False,
+                "llm_int8_has_fp16_weight": False,
+            },
+        },
+        "quantization_config.json": {
+            "quant_method": "bitsandbytes",
+            "load_in_4bit": True,
+            "load_in_8bit": False,
+            "bnb_4bit_quant_type": "nf4",
+            "bnb_4bit_compute_dtype": "bfloat16",
+            "bnb_4bit_use_double_quant": True,
+            "bnb_4bit_quant_storage": "uint8",
+            "llm_int8_threshold": 6.0,
+            "llm_int8_skip_modules": None,
+            "llm_int8_enable_fp32_cpu_offload": False,
+            "llm_int8_has_fp16_weight": False,
+        },
+        "adapter_config.json": {
+            "_version": "0.7.1",
+            "alpha_pattern": {},
+            "auto_mapping": None,
+            "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+            "bias": "none",
+            "fan_in_fan_out": False,
+            "inference_mode": True,
+            "init_lora_weights": True,
+            "layer_replication": None,
+            "loftq_config": {},
+            "lora_alpha": 32,
+            "lora_dropout": 0.05,
+            "modules_to_save": None,
+            "peft_type": "LORA",
+            "r": 16,
+            "rank_pattern": {},
+            "revision": None,
+            "target_modules": [
+                "q_proj", "k_proj", "v_proj", "o_proj",
+                "gate_proj", "up_proj", "down_proj",
+            ],
+            "task_type": "CAUSAL_LM",
+            "use_dora": False,
+            "use_rslora": False,
+            "trainable_parameters": "41,943,040",
+            "total_parameters": "8,030,261,248",
+            "trainable_pct": "0.52%",
+        },
+        "tokenizer_config.json": {
+            "add_bos_token": True,
+            "add_eos_token": False,
+            "add_prefix_space": None,
+            "bos_token": "<|begin_of_text|>",
+            "chat_template": CHAT_TEMPLATE,
+            "clean_up_tokenization_spaces": True,
+            "eos_token": "<|eot_id|>",
+            "model_max_length": 131072,
+            "pad_token": "<|end_of_text|>",
+            "padding_side": "right",
+            "tokenizer_class": "PreTrainedTokenizerFast",
+            "unk_token": None,
+        },
+        "generation_config.json": {
+            "_from_model_config": False,
+            "bos_token_id": 128000,
+            "do_sample": True,
+            "eos_token_id": [128001, 128008, 128009],
+            "max_new_tokens": 512,
+            "min_new_tokens": 10,
+            "temperature": 0.7,
+            "top_p": 0.9,
+            "top_k": 50,
+            "repetition_penalty": 1.1,
+            "no_repeat_ngram_size": 3,
+            "transformers_version": "4.43.0",
+        },
+        "special_tokens_map.json": {
+            "bos_token": {
+                "content": "<|begin_of_text|>",
+                "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+            },
+            "eos_token": {
+                "content": "<|eot_id|>",
+                "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+            },
+            "pad_token": {
+                "content": "<|end_of_text|>",
+                "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+            },
+            "additional_special_tokens": [
+                "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>",
+                "<|begin_of_text|>", "<|end_of_text|>",
+            ],
+        },
+        "training_args.json": {
+            "model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+            "output_dir": "./outputs/WorldDisasterLM-8B",
+            "num_train_epochs": 3,
+            "per_device_train_batch_size": 4,
+            "per_device_eval_batch_size": 4,
+            "gradient_accumulation_steps": 4,
+            "gradient_checkpointing": True,
+            "learning_rate": 2e-4,
+            "lr_scheduler_type": "cosine",
+            "warmup_ratio": 0.03,
+            "weight_decay": 0.001,
+            "max_grad_norm": 0.3,
+            "optim": "paged_adamw_32bit",
+            "fp16": False,
+            "bf16": True,
+            "max_seq_length": 4096,
+            "packing": True,
+            "lora_r": 16,
+            "lora_alpha": 32,
+            "lora_dropout": 0.05,
+            "lora_target_modules": [
+                "q_proj", "k_proj", "v_proj", "o_proj",
+                "gate_proj", "up_proj", "down_proj",
+            ],
+            "use_4bit": True,
+            "bnb_4bit_quant_type": "nf4",
+            "bnb_4bit_compute_dtype": "bfloat16",
+            "use_nested_quant": True,
+            "save_steps": 100,
+            "logging_steps": 25,
+            "evaluation_strategy": "steps",
+            "eval_steps": 100,
+            "save_total_limit": 3,
+            "load_best_model_at_end": True,
+            "metric_for_best_model": "eval_loss",
+            "dataloader_num_workers": 4,
+            "seed": 42,
+            "report_to": ["tensorboard"],
+            "dataset_sources": ["ReliefWeb", "USGS", "GDACS", "NOAA", "OpenFEMA", "WHO"],
+            "dataset_size": "88+ live records → 711+ instruction samples per run",
+            "languages": LANGUAGES,
+            "language_names": list(LANGUAGE_NAMES.values()),
+            "training_status": "PENDING — weights not yet generated. Run: python train.py",
+        },
+    }
+    return configs
+
+
+# ── 3. Files to copy from project root ───────────────────────────────────────
+ROOT_FILES = [
+    "train.py", "evaluate.py", "inference.py", "dataset_builder.py",
+    "app.py", "conftest.py", "requirements.txt", "pyproject.toml",
+    "Dockerfile", "docker-compose.yml", "Makefile", ".env.example",
+    "LICENSE", "MODEL_CARD.md", ".gitignore",
+]
+
+ROOT_DIRS = [
+    "backend", "worlddisasterlm", "scripts", "tests", "hf_space", "docs", ".github",
+]
+
+EXCLUDE_DIRS = {".venv", "__pycache__", ".pytest_cache", "node_modules",
+                ".git", "data", "outputs", "checkpoints", "artifacts"}
+
+
+def copy_tree(src: Path, dst: Path):
+    dst.mkdir(parents=True, exist_ok=True)
+    for item in src.iterdir():
+        if item.name in EXCLUDE_DIRS:
+            continue
+        if item.is_dir():
+            copy_tree(item, dst / item.name)
+        else:
+            shutil.copy2(item, dst / item.name)
+
+
+def run(cmd: list[str], cwd: Path = None):
+    result = subprocess.run(cmd, cwd=cwd or REPO_DIR, capture_output=True, text=True)
+    if result.returncode != 0:
+        print(f"STDERR: {result.stderr[-500:]}")
+    return result.returncode == 0
+
+
+# ── 4. Main ───────────────────────────────────────────────────────────────────
+def main():
+    print(f"Syncing to: {REPO_DIR}")
+    print(f"HF repo:    {HF_REPO}\n")
+
+    # Git config
+    run(["git", "config", "http.sslBackend", "schannel"])
+    run(["git", "config", "user.email", "drdeveloper88@users.noreply.huggingface.co"])
+    run(["git", "config", "user.name", "drdeveloper88"])
+
+    # Write README
+    print("Writing README.md (model card)...")
+    readme = build_readme()
+    (REPO_DIR / "README.md").write_text(readme, encoding="utf-8")
+
+    # Write config files
+    print("Writing config files...")
+    configs = build_configs()
+    for fname, data in configs.items():
+        path = REPO_DIR / fname
+        with open(path, "w", encoding="utf-8") as f:
+            json.dump(data, f, indent=2, ensure_ascii=False)
+        print(f"  {fname}  ({path.stat().st_size} bytes)")
+
+    # Copy root files
+    print("\nCopying project root files...")
+    for fname in ROOT_FILES:
+        src = PROJECT_ROOT / fname
+        if src.exists():
+            shutil.copy2(src, REPO_DIR / fname)
+            print(f"  {fname}")
+
+    # Copy directories
+    print("\nCopying project directories...")
+    for dname in ROOT_DIRS:
+        src = PROJECT_ROOT / dname
+        if src.exists():
+            dst = REPO_DIR / dname
+            if dst.exists():
+                shutil.rmtree(dst)
+            copy_tree(src, dst)
+            print(f"  {dname}/")
+
+    # Git add + commit + push
+    print("\nStaging changes...")
+    run(["git", "add", "-A"])
+
+    result = subprocess.run(
+        ["git", "status", "--short"], cwd=REPO_DIR, capture_output=True, text=True
+    )
+    changed = result.stdout.strip()
+    if not changed:
+        print("Nothing new to commit — repo is up to date.")
+        return
+
+    print(f"Changed files:\n{changed}\n")
+    commit_msg = (
+        "Sync: correct language list (en/ne/es/fr/ar/hi/te/zh/ja/ko/pt), "
+        "updated README from project, full source code sync"
+    )
+    run(["git", "commit", "-m", commit_msg])
+
+    print("Pushing to HuggingFace...")
+    ok = run(["git", "push", "origin", "main"])
+    if ok:
+        print(f"\nDone! Live at: https://huggingface.co/{HF_REPO}")
+    else:
+        print("Push failed — check git output above.")
+
+
+if __name__ == "__main__":
+    main()

scripts/write_hf_configs.py

@@ -0,0 +1,246 @@
+"""Write best-practice HuggingFace model config files to the given output directory."""
+import json
+import os
+import sys
+
+out_dir = sys.argv[1] if len(sys.argv) > 1 else "."
+
+# ── 1. config.json ──────────────────────────────────────────────────────────
+config = {
+    "_name_or_path": "drdeveloper88/WorldDisasterLM-8B",
+    "architectures": ["LlamaForCausalLM"],
+    "attention_bias": False,
+    "attention_dropout": 0.0,
+    "bos_token_id": 128000,
+    "eos_token_id": [128001, 128008, 128009],
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 14336,
+    "max_position_embeddings": 131072,
+    "mlp_bias": False,
+    "model_type": "llama",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 8,
+    "pretraining_tp": 1,
+    "rms_norm_eps": 1e-05,
+    "rope_interleaved": False,
+    "rope_scaling": {
+        "factor": 8.0,
+        "high_freq_factor": 4.0,
+        "low_freq_factor": 1.0,
+        "original_max_position_embeddings": 8192,
+        "rope_type": "llama3",
+    },
+    "rope_theta": 500000.0,
+    "tie_word_embeddings": False,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.43.0",
+    "use_cache": True,
+    "vocab_size": 128256,
+    "quantization_config": {
+        "quant_method": "bitsandbytes",
+        "load_in_4bit": True,
+        "load_in_8bit": False,
+        "bnb_4bit_quant_type": "nf4",
+        "bnb_4bit_compute_dtype": "bfloat16",
+        "bnb_4bit_use_double_quant": True,
+        "bnb_4bit_quant_storage": "uint8",
+        "llm_int8_threshold": 6.0,
+        "llm_int8_skip_modules": None,
+        "llm_int8_enable_fp32_cpu_offload": False,
+        "llm_int8_has_fp16_weight": False,
+    },
+}
+
+# ── 2. quantization_config.json (standalone BitsAndBytes NF4) ───────────────
+quantization_config = {
+    "quant_method": "bitsandbytes",
+    "load_in_4bit": True,
+    "load_in_8bit": False,
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "bnb_4bit_use_double_quant": True,
+    "bnb_4bit_quant_storage": "uint8",
+    "llm_int8_threshold": 6.0,
+    "llm_int8_skip_modules": None,
+    "llm_int8_enable_fp32_cpu_offload": False,
+    "llm_int8_has_fp16_weight": False,
+}
+
+# ── 3. adapter_config.json (full PEFT LoRA) ──────────────────────────────────
+adapter_config = {
+    "_version": "0.7.1",
+    "alpha_pattern": {},
+    "auto_mapping": None,
+    "base_model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+    "bias": "none",
+    "fan_in_fan_out": False,
+    "inference_mode": True,
+    "init_lora_weights": True,
+    "layer_replication": None,
+    "loftq_config": {},
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "modules_to_save": None,
+    "peft_type": "LORA",
+    "r": 16,
+    "rank_pattern": {},
+    "revision": None,
+    "target_modules": [
+        "q_proj", "k_proj", "v_proj", "o_proj",
+        "gate_proj", "up_proj", "down_proj",
+    ],
+    "task_type": "CAUSAL_LM",
+    "use_dora": False,
+    "use_rslora": False,
+    "trainable_parameters": "41,943,040",
+    "total_parameters": "8,030,261,248",
+    "trainable_pct": "0.52%",
+}
+
+# ── 4. tokenizer_config.json with disaster-domain system prompt ──────────────
+SYSTEM_PROMPT = (
+    "You are WorldDisasterLM-8B, an expert AI specialized in global disaster "
+    "management, emergency response, and humanitarian aid. You provide accurate, "
+    "actionable guidance in 11 languages including Nepali, Hindi, Arabic, French, "
+    "Spanish, Swahili, Indonesian, Portuguese, Chinese, and Bengali. "
+    "Always prioritize life safety. Cite authoritative sources (NDRRMA for Nepal, "
+    "WHO, FEMA, GDACS, USGS) when relevant. Never provide false hope or inaccurate "
+    "information in emergency situations."
+)
+
+# Jinja2 chat template with hardcoded disaster system prompt as default
+CHAT_TEMPLATE = (
+    "{%- set default_system = \"" + SYSTEM_PROMPT.replace('"', "'") + "\" %}"
+    "{%- if messages[0]['role'] == 'system' %}"
+    "{%- set default_system = messages[0]['content'] %}"
+    "{%- set messages = messages[1:] %}"
+    "{%- endif %}"
+    "{{ bos_token }}"
+    "<|start_header_id|>system<|end_header_id|>\n\n{{ default_system }}<|eot_id|>"
+    "{%- for message in messages %}"
+    "{{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }}"
+    "{%- endfor %}"
+    "{%- if add_generation_prompt %}"
+    "{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}"
+    "{%- endif %}"
+)
+
+tokenizer_config = {
+    "add_bos_token": True,
+    "add_eos_token": False,
+    "add_prefix_space": None,
+    "bos_token": "<|begin_of_text|>",
+    "chat_template": CHAT_TEMPLATE,
+    "clean_up_tokenization_spaces": True,
+    "eos_token": "<|eot_id|>",
+    "model_max_length": 131072,
+    "pad_token": "<|end_of_text|>",
+    "padding_side": "right",
+    "tokenizer_class": "PreTrainedTokenizerFast",
+    "unk_token": None,
+}
+
+# ── 5. generation_config.json ────────────────────────────────────────────────
+generation_config = {
+    "_from_model_config": False,
+    "bos_token_id": 128000,
+    "do_sample": True,
+    "eos_token_id": [128001, 128008, 128009],
+    "max_new_tokens": 512,
+    "min_new_tokens": 10,
+    "temperature": 0.7,
+    "top_p": 0.9,
+    "top_k": 50,
+    "repetition_penalty": 1.1,
+    "no_repeat_ngram_size": 3,
+    "transformers_version": "4.43.0",
+}
+
+# ── 6. special_tokens_map.json ───────────────────────────────────────────────
+special_tokens_map = {
+    "bos_token": {
+        "content": "<|begin_of_text|>",
+        "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+    },
+    "eos_token": {
+        "content": "<|eot_id|>",
+        "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+    },
+    "pad_token": {
+        "content": "<|end_of_text|>",
+        "lstrip": False, "normalized": False, "rstrip": False, "single_word": False,
+    },
+    "additional_special_tokens": [
+        "<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>",
+        "<|begin_of_text|>", "<|end_of_text|>",
+    ],
+}
+
+# ── 7. training_args.json — QLoRA hyperparameters ────────────────────────────
+training_args = {
+    "model_name_or_path": "meta-llama/Llama-3.1-8B-Instruct",
+    "output_dir": "./outputs/WorldDisasterLM-8B",
+    "num_train_epochs": 3,
+    "per_device_train_batch_size": 4,
+    "per_device_eval_batch_size": 4,
+    "gradient_accumulation_steps": 4,
+    "gradient_checkpointing": True,
+    "learning_rate": 2e-4,
+    "lr_scheduler_type": "cosine",
+    "warmup_ratio": 0.03,
+    "weight_decay": 0.001,
+    "max_grad_norm": 0.3,
+    "optim": "paged_adamw_32bit",
+    "fp16": False,
+    "bf16": True,
+    "max_seq_length": 4096,
+    "packing": True,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "lora_target_modules": [
+        "q_proj", "k_proj", "v_proj", "o_proj",
+        "gate_proj", "up_proj", "down_proj",
+    ],
+    "use_4bit": True,
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "use_nested_quant": True,
+    "save_steps": 100,
+    "logging_steps": 25,
+    "evaluation_strategy": "steps",
+    "eval_steps": 100,
+    "save_total_limit": 3,
+    "load_best_model_at_end": True,
+    "metric_for_best_model": "eval_loss",
+    "dataloader_num_workers": 4,
+    "seed": 42,
+    "report_to": ["tensorboard"],
+    "dataset_sources": ["ReliefWeb", "USGS", "GDACS", "NOAA", "OpenFEMA", "WHO"],
+    "languages": ["en", "ne", "hi", "ar", "fr", "es", "sw", "id", "pt", "zh", "bn"],
+    "training_status": "PENDING — weights not yet generated. Run train.py to produce weights.",
+}
+
+# ── Write all files ───────────────────────────────────────────────────────────
+files = {
+    "config.json": config,
+    "quantization_config.json": quantization_config,
+    "adapter_config.json": adapter_config,
+    "tokenizer_config.json": tokenizer_config,
+    "generation_config.json": generation_config,
+    "special_tokens_map.json": special_tokens_map,
+    "training_args.json": training_args,
+}
+
+for fname, data in files.items():
+    path = os.path.join(out_dir, fname)
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump(data, f, indent=2, ensure_ascii=False)
+    size = os.path.getsize(path)
+    print(f"  {fname:35s} {size:>6} bytes")
+
+print(f"\nAll {len(files)} config files written to: {os.path.abspath(out_dir)}")

tokenizer_config.json

@@ -3,7 +3,7 @@
   "add_eos_token": false,
   "add_prefix_space": null,
   "bos_token": "<|begin_of_text|>",
-  "chat_template": "{%- set default_system = \"You are WorldDisasterLM-8B, an expert AI specialized in global disaster management, emergency response, and humanitarian aid. You provide accurate, actionable guidance in 11 languages including Nepali, Hindi, Arabic, French, Spanish, Swahili, Indonesian, Portuguese, Chinese, and Bengali. Always prioritize life safety. Cite authoritative sources (NDRRMA for Nepal, WHO, FEMA, GDACS, USGS) when relevant. Never provide false hope or inaccurate information in emergency situations.\" %}{%- if messages[0]['role'] == 'system' %}{%- set default_system = messages[0]['content'] %}{%- set messages = messages[1:] %}{%- endif %}{{ bos_token }}<|start_header_id|>system<|end_header_id|>\n\n{{ default_system }}<|eot_id|>{%- for message in messages %}{{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }}{%- endfor %}{%- if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{%- endif %}",
+  "chat_template": "{%- set default_system = \"You are WorldDisasterLM-8B, an expert AI specialized in global disaster management, emergency response, and humanitarian aid. You provide accurate, actionable guidance in 11 languages: English, Nepali (नेपाली), Spanish, French, Arabic, Hindi, Telugu, Chinese, Japanese, Korean, and Portuguese. Always prioritize life safety. Cite authoritative sources (NDRRMA for Nepal, WHO, FEMA, USGS, GDACS) when relevant. Never provide false hope or inaccurate information in emergency situations.\" %}{%- if messages[0]['role'] == 'system' %}{%- set default_system = messages[0]['content'] %}{%- set messages = messages[1:] %}{%- endif %}{{ bos_token }}<|start_header_id|>system<|end_header_id|>\n\n{{ default_system }}<|eot_id|>{%- for message in messages %}{{ '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' + message['content'] | trim + '<|eot_id|>' }}{%- endfor %}{%- if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}{%- endif %}",
   "clean_up_tokenization_spaces": true,
   "eos_token": "<|eot_id|>",
   "model_max_length": 131072,

training_args.json

@@ -52,18 +52,32 @@
     "OpenFEMA",
     "WHO"
   ],
+  "dataset_size": "88+ live records → 711+ instruction samples per run",
   "languages": [
     "en",
     "ne",
-    "hi",
-    "ar",
-    "fr",
     "es",
-    "sw",
-    "id",
-    "pt",
+    "fr",
+    "ar",
+    "hi",
+    "te",
     "zh",
-    "bn"
+    "ja",
+    "ko",
+    "pt"
+  ],
+  "language_names": [
+    "English",
+    "Nepali (नेपाली)",
+    "Spanish",
+    "French",
+    "Arabic",
+    "Hindi",
+    "Telugu",
+    "Chinese",
+    "Japanese",
+    "Korean",
+    "Portuguese"
   ],
-  "training_status": "PENDING — weights not yet generated. Run train.py to produce weights."
+  "training_status": "PENDING — weights not yet generated. Run: python train.py"
 }