hallucinations on empty pages

#4
by elenapop - opened

when passing empty pages (which can happen in documents) to the model, it outputs wild things: lists, latex formulas etc

image

it does not happen only with empty pages, but more complex: photos of documents on a pdf page

LightOn AI org

hello,
could you share your setup(gpu, inference library, sampling params etc) and an example?
we had included empty pages during training exactly for this reason.

NVIDIA GeForce RTX 5090
CUDA Version: 12.8
vllm 0.11.2
vllm serve lightonai/LightOnOCR-2-1B --limit-mm-per-prompt '{"image": 1}' --mm-processor-cache-gb 0 --no-enable-prefix-caching

here is an example of pdf that i split in images and then pass it to the ocr: https://limewire.com/d/qPKsp#qHbLPG7fev (it should be available for one week)

LightOn AI org

thanks for the details!
you can try lightonai/LightOnOCR-2-1B-bbox or lightonai/LightOnOCR-2-1B-base, they don't seem to have this issue!

yes the base one seems better, there is no hallucination in this case, only a explanatory text:

image

LightOn AI org

are you running with same inference parameters as config?
in the demo, i get no output at all for the white page you shared.

yes, the same inference parameters, only the model's name changes; some of the blanc pages get no output and some the text from above

Sign up or log in to comment