amkyawdev commited on
Commit
a90f6d5
Β·
verified Β·
1 Parent(s): 3389b47

Update README

Browse files
Files changed (1) hide show
  1. README.md +135 -0
README.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Myanmar Ghost
3
+ emoji: πŸ‡²πŸ‡²
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: gradio
7
+ sdk_version: 4.0.0
8
+ app_file: app.py
9
+ pinned: false
10
+ ---
11
+
12
+ # Myanmar Ghost
13
+
14
+ Advanced Myanmar Language Understanding Model with Multi-Modal Fusion
15
+
16
+ ## 🎯 Overview
17
+
18
+ Myanmar Ghost is a comprehensive NLP project for Myanmar (Burmese) language understanding, featuring:
19
+
20
+ - **Multi-Modal Data Fusion**: Combines audio prosody with text for sentiment analysis
21
+ - **Active Learning Pipeline**: Efficient annotation with uncertainty sampling
22
+ - **Federated Learning Ready**: Privacy-preserving distributed training
23
+ - **Explainable AI**: SHAP and LIME-based model interpretability
24
+ - **Adversarial Data Augmentation**: Robust training with synonym replacement and perturbation
25
+
26
+ ## πŸ“ Project Structure
27
+
28
+ ```
29
+ Myanmar-Ghost/
30
+ β”œβ”€β”€ configs/ # Configuration files
31
+ β”œβ”€β”€ data/ # Data directory (raw, processed, augmented)
32
+ β”œβ”€β”€ src/
33
+ β”‚ β”œβ”€β”€ data_processing/ # Audio, text processing, multimodal fusion
34
+ β”‚ β”œβ”€β”€ annotation/ # Labeling tools, active learning
35
+ β”‚ β”œβ”€β”€ models/ # Model implementations
36
+ β”‚ β”œβ”€β”€ federated/ # Federated learning components
37
+ β”‚ β”œβ”€β”€ xai/ # Explainable AI modules
38
+ β”‚ β”œβ”€β”€ augmentation/ # Data augmentation
39
+ β”‚ └── utils/ # Utilities
40
+ β”œβ”€β”€ outputs/ # Model outputs, results, logs
41
+ β”œβ”€β”€ tests/ # Unit tests
42
+ └── docs/ # Documentation
43
+ ```
44
+
45
+ ## πŸš€ Quick Start
46
+
47
+ ### Installation
48
+
49
+ ```bash
50
+ pip install -r requirements.txt
51
+ ```
52
+
53
+ ### Data Processing
54
+
55
+ ```bash
56
+ python src/data_processing/audio_processor.py
57
+ python src/data_processing/text_normalizer.py
58
+ ```
59
+
60
+ ### Training
61
+
62
+ ```bash
63
+ python src/models/train.py --config configs/model/transformer_config.yaml
64
+ ```
65
+
66
+ ### Federated Learning
67
+
68
+ ```bash
69
+ # Start server
70
+ python src/federated/server.py --config configs/federated/server_config.yaml
71
+
72
+ # Start client
73
+ python src/federated/client.py --config configs/federated/client_configs/hospital_a.yaml
74
+ ```
75
+
76
+ ## πŸ“Š Features
77
+
78
+ ### 1. Data Versioning (DVC)
79
+
80
+ Track data changes with DVC for reproducibility.
81
+
82
+ ```bash
83
+ dvc add data/processed/v1.2
84
+ git add data/processed/v1.2.dvc
85
+ ```
86
+
87
+ ### 2. Active Learning
88
+
89
+ Select uncertain samples for annotation:
90
+
91
+ ```python
92
+ from src.annotation.active_learning.uncertainty_sampler import UncertaintySampler
93
+
94
+ sampler = UncertaintySampler(model, method="least_confidence")
95
+ selected = sampler.select_samples(samples, tokenizer, n_samples=100)
96
+ ```
97
+
98
+ ### 3. XAI Explanations
99
+
100
+ Understand model decisions:
101
+
102
+ ```python
103
+ from src.xai.shap_explainer import ThankingSHAPExplainer
104
+
105
+ explainer = ThankingSHAPExplainer(model, tokenizer)
106
+ shap_values = explainer.explain("ကျေးဇူးပါ")
107
+ ```
108
+
109
+ ### 4. Data Augmentation
110
+
111
+ ```python
112
+ from src.augmentation.synonym_replacer import MyanmarSynonymReplacer
113
+ from src.augmentation.perturbator import TextPerturbator
114
+
115
+ replacer = MyanmarSynonymReplacer()
116
+ aug_text = replacer.augment_text("ကျေးဇူးပါ", replace_prob=0.3)
117
+ ```
118
+
119
+ ## πŸ“ˆ CI/CD
120
+
121
+ GitHub Actions workflows automatically:
122
+
123
+ - Validate data schemas
124
+ - Check label distributions
125
+ - Run model evaluations
126
+ - Deploy to HuggingFace
127
+
128
+ ## πŸ“ License
129
+
130
+ Apache 2.0 License
131
+
132
+ ## πŸ‘€ Author
133
+
134
+ Aung Myo Kyaw (amkyawdev)
135
+ https://huggingface.co/amkyawdev