The best value in OCR wasn't the smartest model

OCR — having an AI read a paper document or PDF and turn it into text — can now run on large language models (LLMs). So which model, at which setting, gives you the most for your money? We compared several Gemini models, varying how much each one thinks before it answers.

The short version: the best balance of accuracy and cost was not the most capable model. It was a lightweight one — Gemini 3.5 Flash-Lite — run at a medium thinking level. Here is why, step by step.

What we tested

• The material was one business book as a PDF (202 two-page spreads). We scored 60 of them, sampled systematically and without overlap (two sets of 30).
• The accuracy metric is CER (character error rate): after normalizing formatting differences and removing whitespace, the share of characters that are wrong. Lower is better.
• We did not build a hand-checked ground truth. Instead, two different AI models each transcribed the same page in full, and we scored the OCR against each of the two as separate references. On average the two references agreed on about 94 to 95 percent of characters — but that measures how much the two references agree, not the OCR's own accuracy. The two references were never reconciled by a person.

This is one book, under specific conditions. It does not generalize to every kind of document.

The three configurations at a glance

Three representative settings from the comparison (one book, first set of 30):

★ is the value pick. Relative cost is a multiple of Flash-Lite (medium) at 1.0. The lighter Flash-Lite reaches almost the same error rate as the lowest-error setting we found, Gemini 3.6 Flash (0.20 versus 0.19), at about one-seventh the cost.

1. The lighter model was both cheaper and more accurate

Intuition says a heavier model should be more accurate. Yet when we compared the lighter Gemini 3.5 Flash-Lite (medium thinking) against the heavier Gemini 3.5 Flash (low thinking):

• The character error rate fell in both sets — by about 20 percent in one, and about 28 percent in the other.
• Processing cost, meaning what you pay to run the model, fell by just over 70 percent (about 72 to 74 percent) in both sets.

Accuracy went up and cost went down at the same time. "Lighter means giving up accuracy" is not always true. This comparison changes two things at once — a lighter model and a different thinking level — so it weighs whole configurations, not model size alone. Still, the practical result stands, and it is why we treat Flash-Lite as the value anchor. (These cost figures are relative improvements; we leave out absolute amounts.)

2. Why we don't default to the most accurate setting

What if you optimize purely for accuracy? The lowest character error rate in the comparison came from Gemini 3.6 Flash at a medium thinking level, measured on the first set of 30. But that setting costs about seven times as much as the lighter Flash-Lite one.

There are cases where you need the highest accuracy. In this test, though, Flash-Lite gave the better value, and Gemini 3.6 Flash the lowest error rate on the first set. So the split is simple: Flash-Lite for value, Gemini 3.6 Flash when accuracy comes first.

3. More thinking was not simply better

One more finding, about how the thinking level behaves. At first we applied a single "low" setting to every model. Once we measured, that low setting was not the best for either of the two models whose thinking level we varied. For Flash-Lite, low had the most errors; both the minimal and higher settings did better. And which setting was best varied by model, and even by how we measured the error rate.

So it is neither "more thinking is always better" nor "low is enough." You have to measure and choose per model.

Takeaways

Three things came out of this test:

• The best value was the lightweight Gemini 3.5 Flash-Lite (medium thinking) — cheaper than the heavier model, and more accurate.
• On the first set of 30, Gemini 3.6 Flash (medium) had the lowest error rate, at about seven times the cost of Flash-Lite — for accuracy-first work.
• Thinking level is not "higher is better." The sweet spot differs by model.

None of this was visible from spec sheets or reputation. It showed up only when we measured on our own documents. Nihonbashi AI Lab helps with exactly this: evaluating models on your own data and designing how AI fits into the work. To talk it through, get in touch.