bhsinghgrid commited on
Commit
ad411a1
·
verified ·
1 Parent(s): 4864c4b

Update model card: transformer-style usage + local run guide

Browse files
Files changed (2) hide show
  1. LOCAL_SETUP_GUIDE.md +59 -0
  2. README.md +60 -9
LOCAL_SETUP_GUIDE.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Local Setup Guide (Laptop)
2
+
3
+ This model is part of the DevaFlow project (custom D3PM, not native `transformers.AutoModel` format).
4
+
5
+ ## 1) Environment
6
+
7
+ ```bash
8
+ python3.11 -m venv .venv
9
+ source .venv/bin/activate
10
+ pip install -U pip
11
+ pip install -r requirements.txt
12
+ ```
13
+
14
+ ## 2) Quick Inference
15
+
16
+ ```python
17
+ from inference_api import predict
18
+ print(predict("dharmo rakṣati rakṣitaḥ"))
19
+ ```
20
+
21
+ ## 3) Transformer-Style Use
22
+
23
+ ```python
24
+ import torch
25
+ from config import CONFIG
26
+ from inference import load_model, _build_tokenizers
27
+
28
+ cfg = CONFIG
29
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
30
+ model, cfg = load_model("best_model.pt", cfg, device)
31
+ src_tok, tgt_tok = _build_tokenizers(cfg)
32
+
33
+ text = "yadā mano nivarteta viṣayebhyaḥ svabhāvataḥ"
34
+ input_ids = torch.tensor([src_tok.encode(text)], dtype=torch.long, device=device)
35
+ out = model.generate(
36
+ input_ids,
37
+ num_steps=cfg["inference"]["num_steps"],
38
+ temperature=cfg["inference"]["temperature"],
39
+ top_k=cfg["inference"]["top_k"],
40
+ repetition_penalty=cfg["inference"]["repetition_penalty"],
41
+ diversity_penalty=cfg["inference"]["diversity_penalty"],
42
+ )
43
+ ids = [x for x in out[0].tolist() if x > 4]
44
+ print(tgt_tok.decode(ids).strip())
45
+ ```
46
+
47
+ ## 4) Full Project Execution
48
+
49
+ For training, UI, Tasks 1–5, ablation workflow, and HF deployment, use the full project repository and run:
50
+
51
+ - `python train.py`
52
+ - `python inference.py`
53
+ - `python app.py`
54
+ - `python analysis/run_analysis.py --task <1|2|3|4|5|all>`
55
+
56
+ Task 4 note:
57
+ - `--phase generate_configs` first
58
+ - train ablation checkpoints
59
+ - then `--phase analyze`
README.md CHANGED
@@ -1,20 +1,24 @@
1
  ---
2
- datasets:
3
- - paws/sanskrit-verses-gretil
4
  language:
5
  - sa
6
- metrics:
7
- - bleurt
8
- - character
9
- base_model:
10
- - bhsinghgrid/DevaFlow
11
- library_name: adapter-transformers
 
 
12
  ---
13
 
14
  # Sanskrit D3PM Paraphrase Model
15
 
16
  Roman/IAST Sanskrit input to Devanagari output using a D3PM cross-attention model.
17
 
 
 
 
18
  ## Files Included
19
 
20
  - `best_model.pt` — trained checkpoint
@@ -24,6 +28,7 @@ Roman/IAST Sanskrit input to Devanagari output using a D3PM cross-attention mode
24
  - `handler.py` — Hugging Face Endpoint handler
25
  - `model/`, `diffusion/` — architecture modules
26
  - `sanskrit_src_tokenizer.json`, `sanskrit_tgt_tokenizer.json` — tokenizers
 
27
 
28
  ## Quick Local Test
29
 
@@ -32,6 +37,46 @@ from inference_api import predict
32
  print(predict("dharmo rakṣati rakṣitaḥ")["output"])
33
  ```
34
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  ## Endpoint Payload
36
 
37
  ```json
@@ -60,4 +105,10 @@ git remote add origin https://huggingface.co/<your-username>/sanskrit-d3pm
60
  git add .
61
  git commit -m "Initial model release"
62
  git push -u origin main
63
- ```
 
 
 
 
 
 
 
1
  ---
2
+ license: mit
 
3
  language:
4
  - sa
5
+ - en
6
+ tags:
7
+ - sanskrit
8
+ - paraphrase
9
+ - diffusion
10
+ - d3pm
11
+ - pytorch
12
+ pipeline_tag: text-generation
13
  ---
14
 
15
  # Sanskrit D3PM Paraphrase Model
16
 
17
  Roman/IAST Sanskrit input to Devanagari output using a D3PM cross-attention model.
18
 
19
+ This is a **custom PyTorch architecture** (not a native `transformers.AutoModel` checkpoint).
20
+ You can still use it in a transformer-like workflow (load once, pass text, get generated text) via `inference_api.py`.
21
+
22
  ## Files Included
23
 
24
  - `best_model.pt` — trained checkpoint
 
28
  - `handler.py` — Hugging Face Endpoint handler
29
  - `model/`, `diffusion/` — architecture modules
30
  - `sanskrit_src_tokenizer.json`, `sanskrit_tgt_tokenizer.json` — tokenizers
31
+ - `LOCAL_SETUP_GUIDE.md` — full laptop setup and execution guide
32
 
33
  ## Quick Local Test
34
 
 
37
  print(predict("dharmo rakṣati rakṣitaḥ")["output"])
38
  ```
39
 
40
+ ## Transformer-Style Usage (Recommended)
41
+
42
+ Use this model as a reusable generation object:
43
+
44
+ ```python
45
+ import torch
46
+ from config import CONFIG
47
+ from inference import load_model, _build_tokenizers
48
+
49
+ cfg = CONFIG
50
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
51
+
52
+ model, cfg = load_model("best_model.pt", cfg, device)
53
+ src_tok, tgt_tok = _build_tokenizers(cfg)
54
+
55
+ def generate(text: str):
56
+ input_ids = torch.tensor([src_tok.encode(text)], dtype=torch.long, device=device)
57
+ output_ids = model.generate(
58
+ input_ids,
59
+ num_steps=cfg["inference"]["num_steps"],
60
+ temperature=cfg["inference"]["temperature"],
61
+ top_k=cfg["inference"]["top_k"],
62
+ repetition_penalty=cfg["inference"]["repetition_penalty"],
63
+ diversity_penalty=cfg["inference"]["diversity_penalty"],
64
+ )
65
+ ids = [x for x in output_ids[0].tolist() if x > 4]
66
+ return tgt_tok.decode(ids).strip()
67
+
68
+ print(generate("yadā mano nivarteta viṣayebhyaḥ svabhāvataḥ"))
69
+ ```
70
+
71
+ ## About `transformers` Compatibility
72
+
73
+ - This repo does not expose `config.json` + `model.safetensors` in `transformers` format.
74
+ - If you want full `AutoModel`/`pipeline` compatibility, you must create a wrapper architecture and export weights into HF Transformers conventions.
75
+ - For production today, use:
76
+ - `inference_api.py` for Python apps
77
+ - `handler.py` for HF Inference Endpoints
78
+ - `space_repo/app.py` for Gradio UI
79
+
80
  ## Endpoint Payload
81
 
82
  ```json
 
105
  git add .
106
  git commit -m "Initial model release"
107
  git push -u origin main
108
+ ```
109
+
110
+ ## Full Local Laptop Guide
111
+
112
+ For complete setup (training, inference, UI, tasks 1-5, ablation, and deployment), see:
113
+
114
+ - `LOCAL_SETUP_GUIDE.md`