Views
No views yet
deepseek-ai/DeepSeek-OCR on WildKhmerST.
The setting for image processing is the following1base_size=512
2image_size=512
3crop_mode=False1model = FastVisionModel.get_peft_model(
2 model,
3 target_modules = [
4 "q_proj",
5 "k_proj",
6 "v_proj",
7 "o_proj",
8 "gate_proj",
9 "up_proj",
10 "down_proj",
11 ],
12
13 r = 16, # The larger, the higher the accuracy, but might overfit
14 lora_alpha = 16, # Recommended alpha == r at least
15 lora_dropout = 0,
16 bias = "none",
17 random_state = 3407,
18 use_rslora = False, # We support rank stabilized LoRA
19 loftq_config = None, # And LoftQ
20 # target_modules = "all-linear", # Optional now! Can specify a list if needed
21)WildKhmerST is processed to contains pairs of 'text region image' and 'text'.
The processed dataset is uploaded to huggingface platform here.
There are 3 splits. train split is used to fine-tune unsloth/DeepSeek-OCR.
test split is used to evaluate the fine-tuned model.test split.
Concretely, its cer on the whole test split is 237% (result file can be found here)
This is because when the fine-tuned completely misses decoding some images, it hallucinates and then repeats the same text token until the number of maximum tokens is met.
Removing hallucinating cases, cer drops to 67.66% (result file can be found here).