|
|
--- |
|
|
language: |
|
|
- ja |
|
|
- en |
|
|
tags: |
|
|
- xgboost |
|
|
- anime |
|
|
- difficulty-estimation |
|
|
- education |
|
|
- japanese |
|
|
- language-learning |
|
|
- nlp |
|
|
- difficulty-prediction |
|
|
metrics: |
|
|
- rmse |
|
|
- mae |
|
|
- r2 |
|
|
model_type: xgboost |
|
|
--- |
|
|
|
|
|
# Anime Japanese Difficulty Predictor |
|
|
|
|
|
This project implements an XGBoost regression model to predict the Japanese language difficulty of anime series. The model assigns a score on a 0-50 scale based on subtitle linguistics, vocabulary statistics, and semantic content. |
|
|
|
|
|
## Dataset and Ground Truth |
|
|
|
|
|
The model was trained on a dataset of approximately **1,100 anime series and movies**. |
|
|
|
|
|
* **Source:** Difficulty ratings were sourced from **Natively (learnnatively.com)** using the platform's "Data Download" feature. |
|
|
* **Scale:** 0 to 50 (User-generated ratings). |
|
|
* **Distribution:** The dataset is normally distributed but heavily concentrated in the 15-35 range, representing the standard difficulty of most broadcast anime. |
|
|
|
|
|
## Data Collection |
|
|
|
|
|
Subtitle data was aggregated using `jimaku-downloader`, a custom tool that interfaces with the **Jimaku.cc** API. |
|
|
|
|
|
* **Extraction:** The tool utilizes regex-based parsing to identify and map episodes to metadata. |
|
|
* **Selection Logic:** Priority was given to official Web-DL sources and text-based formats (SRT) over OCR or ASS files. |
|
|
* **Potential Noise:** As Jimaku relies on user/group uploads, and episode mapping is automated via regex, the dataset contains a margin of error regarding subtitle timing accuracy and version matching. |
|
|
|
|
|
## Feature Engineering |
|
|
|
|
|
The model utilizes a combination of hard statistical features and semantic embeddings. |
|
|
|
|
|
### 1. Statistical Features |
|
|
* **Density Metrics:** Characters per minute (CPM), Kanji density, Type-Token Ratio (TTR). |
|
|
* **Vocabulary Coverage:** Percentage of words appearing in common frequency lists (Top 1k, 2k, 5k, 10k). |
|
|
* **Comprehension Thresholds:** Number of unique words required to reach 90%, 95%, and 98% text coverage. |
|
|
* **JLPT Distribution:** Proportion of vocabulary corresponding to JLPT levels N5 through N1. |
|
|
* **Part-of-Speech:** Distribution of word types (nouns, verbs, particles, etc.). |
|
|
|
|
|
### 2. Semantic Features |
|
|
* **Text Inputs:** |
|
|
* Series description. |
|
|
* "Lexical Signature": A concatenation of the top 200 most frequent content words (excluding stopwords) extracted from the subtitles. |
|
|
* **Encoding:** Text is encoded using `paraphrase-multilingual-MiniLM-L12-v2`. |
|
|
* **Dimensionality Reduction:** High-dimensional embeddings are reduced to 30 components using PCA. |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
The inference pipeline follows this structure: |
|
|
|
|
|
1. **Preprocessing:** |
|
|
* Numeric features are normalized using `StandardScaler`. |
|
|
* Text inputs are vectorized via SentenceTransformer and reduced via PCA. |
|
|
2. **Estimator:** |
|
|
* Algorithm: XGBoost Regressor. |
|
|
* Optimization: Hyperparameters tuned via Optuna (50 trials) minimizing RMSE. |
|
|
* Validation: 5-Fold Cross-Validation. |
|
|
|
|
|
## Performance |
|
|
|
|
|
Evaluated on a held-out test set (20% split): |
|
|
|
|
|
| Metric | Value | |
|
|
| :------- | :--------- | |
|
|
| **RMSE** | **2.3633** | |
|
|
| **MAE** | **1.8670** | |
|
|
| **R²** | **0.5813** | |
|
|
|
|
|
## Limitations |
|
|
|
|
|
* **Subtitle Quality:** Reliance on user-uploaded subtitles introduces potential variance in transcription accuracy and timing. |
|
|
* **Ground Truth Subjectivity:** Natively ratings are based on user perception of difficulty rather than a standardized linguistic index. |
|
|
* **Parsing Errors:** The automated episode detection in the data collection phase may have resulted in mismatched subtitles for a small fraction of the training data. |
|
|
|
|
|
## Artifacts |
|
|
|
|
|
The trained model is serialized as `anime_difficulty_model.pkl`. This file contains a dictionary with the following keys: |
|
|
* `model`: The trained XGBoost regressor. |
|
|
* `scaler`: Fitted StandardScaler for numeric features. |
|
|
* `pca`: Fitted PCA object for text embeddings. |
|
|
* `feature_cols`: List of numeric column names expected by the pipeline. |
|
|
|
|
|
**Note:** The SentenceTransformer model is not pickled due to size; it must be re-initialized during inference. |
|
|
|
|
|
## Acknowledgements |
|
|
|
|
|
* **Natively:** For providing the difficulty rating dataset. |
|
|
* **Jimaku.cc:** For providing access to the subtitle repository. |
