OCR From Pixels to Text

OCR reads text out of photos in two moves: find where the words sit, then decode what the letters say, surviving fonts, blur and bent pages along the way.

OCR boxes each word first and then decodes each box into characters, so one substitution in five letters reads as a 20 percent character error rate.

Why Does This Exist?

Invoices, street signs, license plates and handwritten forms lock their meaning in pixels, invisible to search and databases. Typing it all by hand does not scale, and classic template matching shatters on the first unfamiliar font. Optical character recognition exists to convert pictures of text into strings machines can store and query.

The field splits into text detection, finding the words, and text recognition, reading them; the general box machinery underneath is object detection. This page covers how the two stages divide the work, how sequence decoding reads a strip, and how error rates are reported.

The one idea you need first. Detection outputs boxes, not letters. Everything downstream reads pixels inside those boxes, so a clipped or tilted box guarantees a garbled string no matter how strong the reader is.

Think of It Like This

A librarian with sticky notes

A librarian facing a wall of framed posters first sticks a note on every frame that holds writing, then sits down with each marked frame and transcribes it letter by letter. She never transcribes unmarked wall, and a note stuck halfway across a frame guarantees a half-missing transcription.

Detection is the sticky-note round and recognition is the transcription round. Where it stops: the librarian rereads a doubtful word in context, while a basic pipeline decodes each strip once, left to right, with no second glance.

How It Actually Works

Detect, rectify, decode

Detection networks propose word or line boxes, often rotated rectangles or polygons, because axis-aligned boxes cannot fit curved shop signs or perspective-slanted pages. Each crop is rectified, meaning warped into a straight horizontal strip, so the reader always sees normalized text. A recognition network then reads the strip: a convolutional backbone extracts a left-to-right feature sequence, a sequence model adds context across positions, and a decoder emits characters. The common decoders are CTC (connectionist temporal classification, which aligns unsegmented frames to letters with a blank symbol) and attention decoding (which glances back at strip positions per emitted character). Document OCR adds layout analysis first to recover columns and tables; scene-text OCR leans on large pretraining to survive wild fonts.

Count errors at character level

Recognizer quality is reported with the character error rate. Let $S$ be substitutions, $D$ deletions, $I$ insertions and $N$ the reference length in characters:

$\mathrm{CER} = \frac{S + D + I}{N}$

Word accuracy calls a whole word wrong over a single letter, so papers report both numbers side by side: the character rate shows the model nearly reads, while word accuracy shows whether the user still retypes.

Worked error rate

A recognizer reads the reference "hello" as "hallo". The first, third, fourth and fifth letters match; the second is one substitution, with no deletions or insertions. So $S = 1$ , $D = 0$ , $I = 0$ , $N = 5$ :

$\mathrm{CER} = \frac{1 + 0 + 0}{5} = 0.2 = 20\%$

Word accuracy scores the same output as simply wrong. Pipelines budget detection and recognition errors separately, because a missed box and a misread letter need different fixes.

Code

1def edit(a, b):2    dp = list(range(len(b) + 1))3    for i, ca in enumerate(a, 1):4        row = [i]5        for j, cb in enumerate(b, 1):6            row.append(min(dp[j] + 1, row[-1] + 1, dp[j - 1] + (ca != cb)))7        dp = row8    return dp[-1]9
10ref, hyp = "hello", "hallo"11print(edit(ref, hyp), edit(ref, hyp) / len(ref))12# -> 1 0.2

Watch Out For

Evaluating on clean scans, deploying on phone photos

Scanned benchmarks flatter models with flat lighting and straight lines, while phone captures bring glare, perspective and motion blur. The symptom is a steep accuracy drop on launch day. Fix it by benchmarking on captures from the actual devices and augmenting training with perspective, blur and glare.

Straight boxes clipping bent text

Axis-aligned boxes slice the first and last letters off curved or rotated words. The symptom is confident readings that always miss edge characters. Fix it with rotated or polygon boxes plus a rectification step before recognition.

The Quick Version

OCR is detection (where the words sit) followed by recognition (what they say).
Detection must handle rotation and curvature, not just straight boxes.
Rectified strips are read by sequence models with CTC or attention decoding.
Character error rate $(S + D + I) / N$ separates near-misses from failures; word accuracy does not.
Phone-photo conditions, not clean scans, decide whether a system survives deployment.