Text Detection in Images

Text detection draws boxes around words in photos, coping with rotation, curvature and dense lines that ordinary object boxes cannot fit.

Text detectors paint curved words pixel by pixel, then fit polygons that tight-box the text where straight boxes swallow background.

Why Does This Exist?

Words in the wild refuse to sit still: shop signs tilt, bottle labels curve, and receipts pack lines edge to edge. Axis-aligned boxes from standard object detection either clip letters or swallow neighbours, and every clipped or merged crop poisons the recognizer downstream. Text detection exists as the specialized first half of optical character recognition: find the words in whatever shape they take, so recognition receives clean crops.

This page covers the two detector families, a hand-computed rotated-box example, and the grouping thresholds that decide real quality.

Think of It Like This

Flexible tape on a curved bottle

Highlighting a straight textbook line takes one swipe of a flat marker. Highlighting ingredients around a curved bottle takes flexible tape that bends with the surface and never covers two lines at once. Segmentation detectors lay that tape: they paint the text region pixel by pixel, bending for free, then polygon fitting cuts the tape outline.

The analogy stops at the learning. Tape follows your hand, while detectors learn text texture from thousands of labelled signs, which is why thin or rare scripts need their own training data.

How It Actually Works

Two families, one job

Regression detectors extend anchor-based detection with rotated boxes $(x, y, w, h, \theta)$ : center, width, height, plus rotation angle. They are fast on straight signage. Segmentation detectors classify each pixel as text or background into a probability map $P$ , then group pixels into instances: DBNet learns a per-pixel threshold map and binarizes differentiably, CRAFT links characters through affinity scores, and shrink-expand grouping keeps high-confidence kernels as seeds before growing them to full words. Both families output rotated boxes or polygons that a rectification step warps straight before recognition.

Worked rotated-box numbers

Take a word that fits a tight 100 by 20 rotated box, area $2000$ , tilted at $\theta = 30$ degrees. An axis-aligned detector must instead return the bounding box of that rotated rectangle: width $100 \times \cos 30 + 20 \times \sin 30 = 86.6 + 10 = 96.6$ , height $100 \times \sin 30 + 20 \times \cos 30 = 50 + 17.3 = 67.3$ , area $\approx 6503$ . Same word, 3.25 times the pixels, most of them background or neighbours. That factor is why wild text needs rotation and polygons rather than straight boxes.

Grouping decides the crops

A segmentation head marks a curved sign with 1240 text pixels at threshold 0.5. Shrink-expand grouping first keeps the 800 high-confidence core pixels at threshold 0.8 as two separate seeds, then grows each seed outward to the full 1240, splitting what one loose threshold would have merged into a single blob. The two resulting polygons become two clean recognition crops instead of one garbled line. Tune these thresholds on your densest documents, not your cleanest signs.

Code

1import math2
3
4def tight_vs_axis_aligned(w: float, h: float, theta_deg: float) -> list:5    """Compare a tight rotated word box against its axis-aligned bounding box."""6    t = math.radians(theta_deg)7    bb_w = w * abs(math.cos(t)) + h * abs(math.sin(t))8    bb_h = w * abs(math.sin(t)) + h * abs(math.cos(t))9    return [round(w * h, 1), round(bb_w * bb_h, 1)]10
11
12print(tight_vs_axis_aligned(100, 20, 30))13# -> [2000, 6503.3]14print(tight_vs_axis_aligned(100, 20, 0))15# -> [2000, 2000.0]

Watch Out For

One threshold for sparse signs and dense receipts

A single binarization threshold that suits isolated shop signs merges packed receipt lines into stripes. The symptom is perfect demo photos beside fused lines on real documents.

Fix it with shrink-expand grouping or per-image adaptive thresholds instead of one global value, and test on the densest page in the batch.

Tiny text downsampled into nothing

Backbones that stride 32 turn an 8-pixel-tall word into a quarter of a feature cell: the text signal is gone before the detection head ever sees it. Small print silently vanishes while headline text scores perfectly.

Keep high-resolution feature maps or tile the input for small text, and measure recall separately by text height instead of averaging over the page.

The Quick Version

Text detection feeds recognition, so its misses and merges cap the whole OCR pipeline.
Regression heads are fast on straight text; segmentation heads win on curves and dense lines.
A 100 by 20 word at 30 degrees needs 6503 straight-box pixels against 2000 tight ones.
Pixel grouping with shrink-expand seeds, not one threshold, separates neighbouring words.
Tune thresholds and resolution on the smallest, densest text you must read.