| # DLNLP - Vietnamese Hate Speech Detection |
|
|
| This repository contains a local notebook workflow for Vietnamese hate speech detection. |
|
|
| ## Project Structure |
|
|
| - `data/raw`: raw CSV files |
| - `data/processed`: cleaned CSV files generated by preprocessing |
| - `notebooks`: notebook pipeline |
| - `outputs/figures`: saved charts and confusion matrices |
| - `outputs/models`: trained model artifacts |
| - `outputs/results`: metrics, reports, and prediction files |
|
|
| ## Dữ liệu đầu vào |
|
|
| The preprocessing notebook expects these files in `data/raw`: |
|
|
| - `train_raw.csv` |
| - `val_raw.csv` |
| - `test_raw.csv` |
|
|
| Each file should contain at least: |
|
|
| - `free_text` |
| - `label_id` |
|
|
| ## Setup |
|
|
| Create and activate a virtual environment, then install dependencies: |
|
|
| ```bash |
| python -m venv .venv |
| .venv\Scripts\activate |
| pip install -r requirements.txt |
| ``` |
|
|
| If you already have a working Python environment, installing from `requirements.txt` is enough. |
|
|
| ## How to run locally |
|
|
| Run the notebooks in this order: |
|
|
| 1. `notebooks/01_data_preprocessing.ipynb` |
| 2. `notebooks/02_baseline_tfidf_svm.ipynb` |
| 3. `notebooks/03_bilstm_train_evaluate.ipynb` |
| 4. `notebooks/04_phobert_train_evaluate.ipynb` |
| 5. `notebooks/05_results_error_analysis.ipynb` |
|
|
| ## Demo giao diện |
|
|
| Sau khi đã có các file model trong `outputs/models`, chạy giao diện demo bằng: |
|
|
| ```bash |
| streamlit run app.py |
| ``` |
|
|
| ## Notes |
|
|
| - To make the repository clone-and-run for the demo app, keep the trained artifacts in `outputs/models`. |
| - Do not commit local environment folders such as `.venv`, notebook caches, or temporary training checkpoints. |
| - The notebooks resolve the project root automatically from the local workspace, so Colab Drive mounting is not required. |
| - Notebook 1 creates `data/processed` from the raw CSV files. |
| - Notebooks 2, 3, and 4 write outputs into `outputs/models`, `outputs/results`, and `outputs/figures`. |
| - Notebook 5 reads the artifacts from those local output folders and performs comparison/error analysis. |
| - For notebook 3 and notebook 4, the package list includes `pyvi`, `transformers`, `torch`, and `accelerate`. |
|
|
| ## Expected outputs |
|
|
| After a full run, you should see files such as: |
|
|
| - `data/processed/train_processed.csv` |
| - `outputs/models/svm_model.joblib` |
| - `outputs/models/bilstm_best.pt` |
| - `outputs/models/phobert_base/` |
| - `outputs/results/svm_metrics.csv` |
| - `outputs/results/bilstm_metrics.csv` |
| - `outputs/results/phobert_metrics.csv` |
| - `outputs/results/phobert_test_predictions.csv` |
| - `outputs/figures/*.png` |
|
|
