Could you share the training source code for the ChineseErrorDetectorElectra?
Hi, and thanks for open-sourcing this Chinese error pre-filter model β the results look great π
I'm trying to reproduce the char-level TokenClassification training locally, following the approach described in the README, so I can verify my own training setup against yours. However, this repo only ships the model weights, data samples (cec_train_char_sample_50k.jsonl, etc.) and the docs β the training/eval scripts repeatedly referenced in the README are not included, e.g.:
train_char.py (CharTokenTrainer, class weights / Focal loss / neg_subsample_keep, β¦)
char_align.py (char-level labeling via difflib.SequenceMatcher)
evaluate_char.py (--by_task, --sweep_sentence)
pipeline_char.py / char_inference.py
prepare_task_validation_jsonl.py and the rest of build_hf_aligned_jsonl.py's pipeline
the command docs TRAIN_CHAR_ELECTRA.txt, UPLOAD_AND_TRAIN.txt, CHAR_PIPELINE.txt
Would you be able to:
1.Publish these training/eval scripts β e.g. in a separate GitHub repo, or added to this HF repo?
2.If full release isn't possible right now, could you at least share the char alignment/labeling logic (char_align.py) and the full training hyperparameters (loss type, class weights, LR, epochs, batch size, max_length, best-metric selection, beyond what's in training_char_meta.json)? Those two matter most for reproduction.
My only goal is to validate the correctness of my local training. Thanks a lot!