Kimang18/deepseek_ocr_Khmer_finetuned

Objective

Fine-tune deepseek-ai/DeepSeek-OCR on WildKhmerST. The setting for image processing is the following

base_size=512
image_size=512
crop_mode=False

The setting for LoRA is the following

model = FastVisionModel.get_peft_model(
    model,
    target_modules = [
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],

    r = 16,           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 16,  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
    # target_modules = "all-linear", # Optional now! Can specify a list if needed
)

For future benchmarking in Khmer OCR task, WildKhmerST is processed to contains pairs of 'text region image' and 'text'. The processed dataset is uploaded to huggingface platform here. There are 3 splits. train split is used to fine-tune unsloth/DeepSeek-OCR. test split is used to evaluate the fine-tuned model.

The fine-tuned model performs better on Khmer text, yet it still completely misses some images in the test split. Concretely, its cer on the whole test split is 237% (result file can be found here) This is because when the fine-tuned completely misses decoding some images, it hallucinates and then repeats the same text token until the number of maximum tokens is met. Removing hallucinating cases, cer drops to 67.66% (result file can be found here).

Uploaded finetuned model

• Developed by: Kimang18
• License: apache-2.0
• Finetuned from model : deepseek-ai/DeepSeek-OCR

This deepseek_vl_v2 model was trained 2x faster with Unsloth and Huggingface's TRL library.

Usage

TBA