ProjectsDocument understandingSmall VLMs

MiniVLMDocEval

Which sub-1B vision-language model reads documents best, and where they all break

Six open-source models from 0.45B to 0.9B parameters, five document benchmarks from reading a word to looking up a table cell, and one matched protocol, so the scores compare.

A ChartQA bar chart of emotionsNumeric

What was the 4th most popular emotion?

Gold answerInspired
Qwen3.5-0.8BInspired

Document understanding is a natural job for on-device vision-language models, yet models under a billion parameters are barely characterized on it. MiniVLMDocEval scores six of them on five benchmarks arranged as a ladder, from recognizing text (OCRBench) through extraction (DocVQA), numeric reasoning (ChartQA) and layout (InfoVQA) to tabular reasoning (TableVQA), with matched decoding, precision and samples.

Qwen3.5-0.8B is the strongest foundation, with a mean of 70.3 and three of five benchmarks won. Within this size band architecture beats parameter count: the 0.45B LFM2.5-VL outranks both 0.9B models and is the fastest. The image-resolution strategy, not scale, drives both accuracy and latency. And no model clears 50.1 on visual tables, where the best model's weakness turns out to be plain lookup, not financial numbers.

70.3

mean score of Qwen3.5-0.8B, the strongest foundation, best on DocVQA, InfoVQA and TableVQA.

N = 1,000 per benchmark, seed 42
0.45B

LFM2.5-VL, the smallest model, beats both 0.9B models on ChartQA (73.1) and is the fastest at 0.24 s per sample.

Architecture over parameter count
50.1

is the best TableVQA score of any model: visual table question answering is the frontier of this size band.

TableVQA-Bench, 1,000 questions
27.8 / 84.0

for the best model on Wikipedia-style table lookup against financial tables: the gap is lookup, not numbers.

TableVQA sub-domains

The leaderboard

Every model ran under the same protocol: greedy decoding, 16-bit weights, a 128-token cap and the same 1,000 questions per benchmark. Scores are each benchmark's own metric on a 0 to 100 scale. Sort by any column; the shading follows the score within each benchmark.

Table 1.Accuracy on the five benchmarks and their mean, with the mean seconds per sample. Latency is indicative: it was measured across GPU types, and its ordering matters more than the exact values.

Resolution strategy, not size, sets the speed

LFM2.5-VL caps images at 512 pixels a side and answers in a quarter of a second; FastVLM compresses high-resolution images into few tokens yet takes almost four seconds, the slowest in the field. LFM2.5-VL and Qwen3.5 are the two ends of the front: no model is both faster and more accurate than either. Switch the benchmark to see where the front moves.

Pareto front
Figure 1.Accuracy against mean seconds per sample, on a log scale, for each model. The dashed line joins the models that no other model beats on both. Point at a model for its numbers.

The table gap is visual lookup

Breaking TableVQA into its four sub-domains turns a vague weakness into a precise one. Qwen3.5-0.8B is already strong on financial tables, with their numbers and units, and fair on true-or-false fact checks. It fails at plain lookup over dense Wikipedia-style tables, the largest slice of the benchmark. That runs against the usual assumption in table question answering, which treats numeric tables as the hard case.

VWTQLookup, real Wikipedia tables
27.8
VWTQ-SynLookup, synthetic Wikipedia tables
33.1
VTabFactTrue or false fact checks
55.5
FinTabNetQAFinancial tables, numbers and units
84.0
Figure 2.Qwen3.5-0.8B's accuracy on each TableVQA sub-domain. VWTQ is about half of the benchmark, so it also carries the most weight in the aggregate.

The misses are lookup mistakes, not failures to read: miscounting rows, answering from the neighbouring row or column, or returning a cell of the wrong kind, such as a year where a count was asked.

    Figure 3.Six of Qwen3.5-0.8B's errors on Wikipedia-style tables, with the gold answer in green and the model's in red. Choose a question to see its table.

    A fine-tune the gate rejected

    The second part aimed one LoRA fine-tune at the measured gap, with the success criterion fixed before any training: TableVQA had to rise by at least 3 points, with the gain in the lookup sub-domains, and no other benchmark could fall by more than 1.5. It failed. Only VTabFact, whose answers are just true or false, held up; everything that must name a specific value collapsed.

    Figure 4.Scores before and after the fine-tune on the same seed-42 subsets, against the rule fixed in advance. Sub-domains are shown for diagnosis and carry no rule of their own.

    The cause is a mismatch in answer style. The training set gives verbose, full-sentence answers, while every benchmark scores a short cell value or span, so the model learned to talk and exact-match scoring punished it everywhere. The gate did its job: it rejected a degraded model instead of letting a misleading gain through. The lookup gap is still open, and the corrected recipe is short-answer supervision that matches the benchmarks. The dataset, the gate and the failed adapter are public.

    Models and protocol

    Six generalist models under a billion parameters, from five labs, chosen to span the main ways of handling high-resolution pages: native dynamic resolution, tiling, crops and token compression. Three of them, FastVLM, Qwen3.5 and LFM2.5-VL, needed new wrappers in VLMEvalKit, which the repository contributes.

    Decoding
    Greedy
    Precision
    16-bit
    Answer cap
    128 tokens
    Questions
    1,000 per benchmark
    ModelLabParamsEncoder and backboneResolution strategy
    Qwen3.5-0.8BAlibaba0.8BQwen-native, gated-delta MoENative dynamic, NaViT-style
    InternVL3-1BOpenGVLab0.9BInternViT-300M + Qwen2.5-0.5BDynamic high-resolution tiling, 448²
    LFM2.5-VL-450MLiquid AI0.45BSigLIP2-86M + LFM2-350MNative up to 512², tunable tiling
    LLaVA-OneVision-0.5BLMMs-Lab0.9BSigLIP + Qwen2-0.5BAnyRes multi-crop
    SmolVLM2-500MHugging Face0.5BSigLIP + SmolLM2-360MSplit images, pixel-shuffle compression
    FastVLM-0.5BApple0.8BFastViTHD + Qwen2-0.5BHigh resolution, few-token compression
    Table 2.The six models. Parameter counts include the vision encoder.

    Half precision costs nothing here. On SmolVLM2 and 100 OCRBench questions, fp16 scored 62.0 against 57.0 in fp32 and cut generation time from 209 to 116 seconds. Raising the answer cap from 128 to 2,048 tokens changed no score on DocVQA or InfoVQA.

    Citation

    Please cite this work as

    @misc{pakdamansavoji2026minivlmdoceval,
      title        = {Sub-1B VLM for Document Understanding: A Systematic Evaluation and a Targeted Improvement},
      author       = {Pakdamansavoji, Sajjad},
      year         = {2026},
      howpublished = {\url{https://github.com/forall-labs/MiniVLMDocEval}}
    }

    Reproducing

    Six models need two incompatible versions of transformers, so a run has two phases and a restart between them. Run the Colab notebook top to bottom, or scripts/reproduce.sh on any CUDA machine. Everything is seeded and pinned, and every prediction, score and log is in the result archive.

    Released

    The training set and the adapter that failed the gate on Hugging Face, the technical report, and a thread on X.