mfeb

/

albert-xxlarge-v2-squad2

like 2

I got the following accuracy 88.09 / 84.83 (F1/Exact) with this checkpoint. The reported accuracy in the original repo is 89.8/86.9.

I used the following command to measure accuracy.

python3 transformer-sparsity/examples/pytorch/question-answering/run_qa.py \
                        --model_name_or_path ${model} \
                        --dataset_name squad_v2 \
                        --do_eval \
                        --per_device_eval_batch_size 8 \
                        --max_seq_length 512 \
                        --doc_stride 128 \
                        --fp16 \
                        --fp16_opt_level O1 \
                        --version_2_with_negative \
                        --overwrite_output_dir \
                        --output_dir ${ckpt_path} 2>&1 | tee ~/${ckpt_path}/eval_run_$(date +"%Y_%m_%d_%I_%M_%p").log