Paper breakdown

LLaVA

The 2023 paper that demonstrated how to turn an open-source text LLM into a powerful multimodal model by projecting visual features into the text token space.

Paper: Visual Instruction Tuning

Authors: Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee · 2023

Read the paper

LLaVA connects a frozen vision encoder to a frozen LLM using a simple linear projection matrix, translating image patches into 'visual tokens' the LLM can read.

The Problem

By early 2023, open-source language models like LLaMA were becoming highly capable at following text instructions, but they were blind. Proprietary models like GPT-4V could "see" and reason about images, but how they were built was secret. Researchers needed a cheap, effective way to graft visual capabilities onto existing, pre-trained text-only LLMs without destroying their language abilities.

The Idea

The authors created LLaVA (Large Language-and-Vision Assistant). The key idea was extreme simplicity: take a pre-trained vision model (CLIP), take a pre-trained text model (LLaMA), and connect them with a single trainable linear matrix. This matrix acts as a "translator," converting the visual embeddings from CLIP into the exact same vector space as LLaMA's word embeddings. To the LLM, the image simply looks like a sequence of strange new words.

How It Works

The LLaVA architecture consists of three parts:

Vision Encoder: A frozen CLIP ViT model. It converts an image into a sequence of patch embeddings.
The Connector: A trainable linear projection layer (just a matrix multiplication). It projects the visual embeddings into the exact dimensionality of the LLM's text embeddings.
The LLM: A LLaMA model that receives a sequence of tokens. The sequence contains the projected "visual tokens" followed by standard text tokens (the user's prompt).

The training happens in two stages:

Stage 1 (Pre-training for Feature Alignment): Both the vision encoder and the LLM are frozen. Only the linear connector is trained to predict text captions for images. This teaches the connector to translate visual features into text space.
Stage 2 (Fine-tuning End-to-End): The visual encoder stays frozen, but the LLM and the connector are trained on a dataset of "visual instructions" (generated by GPT-4) to teach the model how to follow commands about images.

Why It Mattered

LLaVA proved that creating a multimodal LLM did not require training a massive network from scratch. By standing on the shoulders of CLIP and LLaMA, a highly capable multimodal assistant could be trained on academic budgets (it cost less than $100 to train). It became the blueprint for almost all open-weights multimodal models.

What Came After

LLaVA 1.5 replaced the linear projection with a slightly more expressive MLP and swapped CLIP for SigLIP, drastically improving performance. The "connector" paradigm became standard, eventually evolving to handle interleaved text and images, video frames, and multiple image resolutions (as seen in models like Qwen-VL and later LLaVA versions).