test

Fix a bug

Use the original training tokenizer instead of building tokenizer from test data

Use the original training tokenizer instead of creating tokenizer from scratch

Test

Debug tokenizer import

Debug tokenizer import

Debug tokenizer import

Update tokenizer class

Test

Debug

Debug

Idk what error

Edit mismatching vocab size leading for mismatching embedding size for training and inference

Edit error parameters

Save vocab size with model_state_dict

Edit default mismatch in LSTM training and inference

Edit some saving and loading details

Edit error parameters

Edit error parameters in inference_lstm.py

Edit error parameters in inference_lstm.py

Edit error parameters in inference_lstm.py

Edit some typos

Add test dataset and example inference/evaluation for LSTM model

Update example use and remove unused class

Add LSTM fine tuning

Add gitignore

Fix a bug

First commit