Gen AI

Vision-Language Models (VLMs)

To give ChatGPT the ability to see, you don't need to rebuild ChatGPT from scratch. You can take a pre-trained Vision model, a pre-trained Text model, and build a tiny 'translator' network between them.

Vision-Language Models fuse a pre-trained Vision Encoder (like CLIP) with a pre-trained LLM, translating images into a language the LLM can understand.

Why Does This Exist?

In 2023, open-source developers faced a problem: OpenAI had just released GPT-4 with Vision (GPT-4V). It was a single, massive multimodal AI that could look at a picture of a broken bicycle and tell you exactly how to fix it.

The open-source community had incredible text models (like Llama) and incredible vision models (like CLIP), but no single model that could do both. Training a massive multimodal model from scratch requires tens of millions of dollars in compute. Instead of starting from scratch, researchers asked a brilliant question: Can we just glue CLIP and Llama together?

The answer was yes. This architecture—connecting a frozen Vision Encoder to a frozen LLM via a tiny translating network—is the defining architecture of modern Vision-Language Models (VLMs) like LLaVA, Qwen-VL, and open-source multimodal systems.

Think of It Like This

The English Professor and the French Art Critic

Imagine you want a brilliant analysis of a French painting.

You have an English Literature Professor (The LLM). They are a genius writer, but they are completely blind.
You have a French Art Critic (CLIP). They can perfectly analyze the painting, but they only speak French.

Instead of sending the Professor to art school for 4 years (training from scratch), you hire a cheap bilingual translator (The Projector). The French Art Critic looks at the painting and dictates their thoughts to the translator. The translator turns those thoughts into English and hands them to the Professor. The Professor uses those notes to write a brilliant, nuanced essay.

How It Actually Works

The architecture of a modern VLM consists of exactly three pieces.

1. The Vision Encoder (The Eyes)

Usually, this is a pre-trained CLIP ViT (Vision Transformer). We freeze the weights of this model so they cannot change. When you upload an image, the Vision Encoder chops it into patches (e.g., 256 patches) and converts each patch into a dense vector (an embedding). At this point, the image is represented as a list of 256 "visual words."

2. The Multimodal Projector (The Translator)

The LLM has absolutely no idea what these visual words mean. Its embedding space is strictly for English text. We insert a tiny neural network (often just a simple Linear layer or a small Multi-Layer Perceptron) called the Projector. Its only job is to translate the visual embeddings into language embeddings. It maps the visual representation of a "dog" into the exact mathematical coordinate where the LLM expects to find the text token "dog".

3. The Large Language Model (The Brain)

Usually, this is a pre-trained model like Llama 3 or Mistral. We also freeze the weights of this model. We take the translated visual tokens from the Projector, concatenate them with the text prompt (e.g., "What is in this image?"), and feed the whole sequence into the LLM. Because the visual tokens have been perfectly translated into the LLM's language space, the LLM just "reads" the image as if it were a highly detailed paragraph of text, and then autoregressively generates its response.

4. Training (Visual Instruction Tuning)

Because we freeze both the massive Vision Encoder and the massive LLM, training a VLM is incredibly cheap. The only thing we actually train is the tiny Projector. We train it using Visual Instruction Tuning. We show the model an image and a question, and use backpropagation to update only the Projector's weights until the LLM outputs the correct answer. This allows researchers to build state-of-the-art VLMs for a few hundred dollars instead of tens of millions.

Show Me the Code

This pseudocode shows the data flow of a VLM. Notice how the Projector acts as the crucial bridge between the two massive, frozen models.

1import torch2import torch.nn as nn3
4class VisionLanguageModel(nn.Module):5    def __init__(self, clip_model, llama_model):6        super().__init__()7        # 1. The massive frozen models8        self.vision_encoder = clip_model.vision_model9        self.llm = llama_model10        11        # 2. The tiny, trainable Projector (Translator)12        # Maps CLIP's 768-dimension space to Llama's 4096-dimension space13        self.projector = nn.Linear(in_features=768, out_features=4096)14        15    def forward(self, image_pixels, text_prompt_tokens):16        # 3. Extract visual features (e.g. 256 patches)17        # Shape: (Batch, 256, 768)18        with torch.no_grad():19            visual_features = self.vision_encoder(image_pixels)20            21        # 4. Translate visual features into LLM language space22        # Shape: (Batch, 256, 4096)23        translated_visual_tokens = self.projector(visual_features)24        25        # 5. Get the standard text embeddings26        # Shape: (Batch, Prompt_Length, 4096)27        text_embeddings = self.llm.get_input_embeddings()(text_prompt_tokens)28        29        # 6. Concatenate them together! 30        # The LLM now just sees one long sequence of 4096-dimension tokens31        combined_input = torch.cat([translated_visual_tokens, text_embeddings], dim=1)32        33        # 7. Generate the answer34        output = self.llm.generate(inputs_embeds=combined_input)35        return output

Watch Out For

The Blindness to Details

Because the Vision Encoder (CLIP) compresses the image heavily, and the Projector translates it into a very broad "language concept," standard VLMs are notoriously bad at fine-grained details. If you show a VLM a picture of a receipt and ask it to read the tax amount, it will often fail or hallucinate a number. It can see the "concept" of a receipt, but it cannot literally "read" the pixels. This has led to newer architectures bypassing CLIP entirely for document-understanding tasks.

The Quick Version

Training a massive multimodal AI from scratch is prohibitively expensive.
Modern Vision-Language Models (VLMs, like LLaVA) solve this by gluing an existing Vision Encoder (like CLIP) to an existing LLM (like Llama).
The bridge between the two models is a tiny neural network called the Projector.
The Projector's job is to translate the dense visual embeddings into language embeddings, allowing the LLM to "read" the image as if it were text.
Because only the tiny Projector is trained (and the massive LLM and Vision Encoder are frozen), building powerful open-source VLMs is incredibly cheap and efficient.