ProjectsDocument understandingSmall VLMs
MiniVLMDocEval
Which sub-1B vision-language model reads documents best, and where they all break
Six open-source models from 0.45B to 0.9B parameters, five document benchmarks from reading a word to looking up a table cell, and one matched protocol, so the scores compare.
NumericWhat was the 4th most popular emotion?
Document understanding is a natural job for on-device vision-language models, yet models under a billion parameters are barely characterized on it. MiniVLMDocEval scores six of them on five benchmarks arranged as a ladder, from recognizing text (OCRBench) through extraction (DocVQA), numeric reasoning (ChartQA) and layout (InfoVQA) to tabular reasoning (TableVQA), with matched decoding, precision and samples.
Qwen3.5-0.8B is the strongest foundation, with a mean of 70.3 and three of five benchmarks won. Within this size band architecture beats parameter count: the 0.45B LFM2.5-VL outranks both 0.9B models and is the fastest. The image-resolution strategy, not scale, drives both accuracy and latency. And no model clears 50.1 on visual tables, where the best model's weakness turns out to be plain lookup, not financial numbers.
mean score of Qwen3.5-0.8B, the strongest foundation, best on DocVQA, InfoVQA and TableVQA.
N = 1,000 per benchmark, seed 42LFM2.5-VL, the smallest model, beats both 0.9B models on ChartQA (73.1) and is the fastest at 0.24 s per sample.
Architecture over parameter countis the best TableVQA score of any model: visual table question answering is the frontier of this size band.
TableVQA-Bench, 1,000 questionsfor the best model on Wikipedia-style table lookup against financial tables: the gap is lookup, not numbers.
TableVQA sub-domainsThe leaderboard
Every model ran under the same protocol: greedy decoding, 16-bit weights, a 128-token cap and the same 1,000 questions per benchmark. Scores are each benchmark's own metric on a 0 to 100 scale. Sort by any column; the shading follows the score within each benchmark.
Resolution strategy, not size, sets the speed
LFM2.5-VL caps images at 512 pixels a side and answers in a quarter of a second; FastVLM compresses high-resolution images into few tokens yet takes almost four seconds, the slowest in the field. LFM2.5-VL and Qwen3.5 are the two ends of the front: no model is both faster and more accurate than either. Switch the benchmark to see where the front moves.
The table gap is visual lookup
Breaking TableVQA into its four sub-domains turns a vague weakness into a precise one. Qwen3.5-0.8B is already strong on financial tables, with their numbers and units, and fair on true-or-false fact checks. It fails at plain lookup over dense Wikipedia-style tables, the largest slice of the benchmark. That runs against the usual assumption in table question answering, which treats numeric tables as the hard case.
The misses are lookup mistakes, not failures to read: miscounting rows, answering from the neighbouring row or column, or returning a cell of the wrong kind, such as a year where a count was asked.
A fine-tune the gate rejected
The second part aimed one LoRA fine-tune at the measured gap, with the success criterion fixed before any training: TableVQA had to rise by at least 3 points, with the gain in the lookup sub-domains, and no other benchmark could fall by more than 1.5. It failed. Only VTabFact, whose answers are just true or false, held up; everything that must name a specific value collapsed.
The cause is a mismatch in answer style. The training set gives verbose, full-sentence answers, while every benchmark scores a short cell value or span, so the model learned to talk and exact-match scoring punished it everywhere. The gate did its job: it rejected a degraded model instead of letting a misleading gain through. The lookup gap is still open, and the corrected recipe is short-answer supervision that matches the benchmarks. The dataset, the gate and the failed adapter are public.
Models and protocol
Six generalist models under a billion parameters, from five labs, chosen to span the main ways of handling high-resolution pages: native dynamic resolution, tiling, crops and token compression. Three of them, FastVLM, Qwen3.5 and LFM2.5-VL, needed new wrappers in VLMEvalKit, which the repository contributes.
- Decoding
- Greedy
- Precision
- 16-bit
- Answer cap
- 128 tokens
- Questions
- 1,000 per benchmark
| Model | Lab | Params | Encoder and backbone | Resolution strategy |
|---|---|---|---|---|
| Qwen3.5-0.8B | Alibaba | 0.8B | Qwen-native, gated-delta MoE | Native dynamic, NaViT-style |
| InternVL3-1B | OpenGVLab | 0.9B | InternViT-300M + Qwen2.5-0.5B | Dynamic high-resolution tiling, 448² |
| LFM2.5-VL-450M | Liquid AI | 0.45B | SigLIP2-86M + LFM2-350M | Native up to 512², tunable tiling |
| LLaVA-OneVision-0.5B | LMMs-Lab | 0.9B | SigLIP + Qwen2-0.5B | AnyRes multi-crop |
| SmolVLM2-500M | Hugging Face | 0.5B | SigLIP + SmolLM2-360M | Split images, pixel-shuffle compression |
| FastVLM-0.5B | Apple | 0.8B | FastViTHD + Qwen2-0.5B | High resolution, few-token compression |
Half precision costs nothing here. On SmolVLM2 and 100 OCRBench questions, fp16 scored 62.0 against 57.0 in fp32 and cut generation time from 209 to 116 seconds. Raising the answer cap from 128 to 2,048 tokens changed no score on DocVQA or InfoVQA.