LLaVA Visual Chat Tuning

Plug CLIP patches into a chat model through one matrix and tune on machine-made instructions. The bot then chats about photos.

LLaVA links CLIP patches to a chat model with one projection

Why Does This Exist?

Chat models read text but not pixels. LLaVA adds vision with the smallest possible joint: a frozen CLIP encoder, one projection matrix $W$ , and a chat LLM. Stage one aligns $W$ on captions; stage two tunes on $158000$ instruction pairs. The result chats about photos, grounding answers in image captioning skill.

This page covers projection plus instruction tuning only. Resampler bridges are a separate design. A patch is one CLIP image token, the projection is $h = W v$ into word space, and an instruction is a photo plus an order plus its answer.

Think of It Like This

A phrasebook plus role-play lessons

A traveler gets a phrasebook mapping foreign picture cards to home words, then takes 158000 role-play lessons ordering food and asking directions. The book is thin, but practice makes chat smooth.

$W$ is the phrasebook, CLIP cards are pictures, and instruction pairs are lessons. Where it stops: phrasebooks stay fixed in print, while $W$ keeps learning in stage two.

How It Actually Works

Encode the photo to patches $V$ with frozen CLIP. Project to $H = W V$ and prepend those visual tokens to the instruction tokens. Train next-token loss on answers only. Stage one uses $595000$ captions to train $W$ alone; stage two uses $158000$ instructions to train $W$ plus the LLM. In plain words: captions teach the mapping, instructions teach the manners, and answers alone carry loss.

Worked example

One patch $v = [0.6, 0.8]$ meets $W = [[1, 0.5], [0.5, 1]]$ . Then $h_1 = 1 \times 0.6 + 0.5 \times 0.8 = 0.6 + 0.4 = 1.0$ and $h_2 = 0.5 \times 0.6 + 1 \times 0.8 = 0.3 + 0.8 = 1.1$ , so $h = [1.0, 1.1]$ . Fed with order tokens, the answer loss falls from $2.4$ nats early to $1.1$ after tuning on the fixture. The drop measures chat fit, since $W$ shapes every visual token.

Code

The point of this snippet is the projection: multiplying the fixture patch by $W$ in plain Python reproduces the hand-computed visual token.

1v = [0.6, 0.8]2W = [[1.0, 0.5], [0.5, 1.0]]3
4h = [round(W[0][0] * v[0] + W[0][1] * v[1], 2),5     round(W[1][0] * v[0] + W[1][1] * v[1], 2)]6print(h)  # -> [1.0, 1.1]

The output $[1.0, 1.1]$ matches the hand computation, confirming each CLIP patch lands in word space through one matrix multiply.

Watch Out For

Trusting machine-made instructions blindly

The $158000$ pairs come from a text model that never saw the photos, so some answers invent objects. Bots then hallucinate with confidence. Filter by matching scores and probe on image retrieval before release.

Shipping the stage-one checkpoint as a chatbot

After stage one $W$ is aligned but the LLM never practiced instructions, so answers read as captions rather than replies. The symptom is accurate descriptions that ignore orders like compare or explain. Fix: run stage two before judging chat quality; caption loss alone never teaches manners.

The Quick Version

LLaVA maps CLIP patches into a chat model with one matrix $W$ .
Stage one aligns $W$ on $595000$ captions; stage two tunes $W$ plus the LLM on $158000$ pairs.
Answers alone carry the loss.
The fixture patch $[0.6, 0.8]$ projects to $[1.0, 1.1]$ .
Machine-made data needs filtering or hallucinations grow.