Data Concepts

Multimodal Data Fusion and Alignment

Fusion is the modelling question and alignment is the data question. Getting the image and the words that truly describe it paired up is where the work is.

Two encoders project into one shared space, and training pushes the similarity of matched pairs up the diagonal of the similarity matrix while pushing everything off it down

Why Does This Exist?

A product-listing model has photographs, titles, descriptions and a category tree. Every modality obviously helps. The team concatenates image features onto text features, trains, and lands almost exactly where the text-only baseline was.

Two things went wrong, and only one of them is about modelling. The image vector had 2048 dimensions against 300 for text, so the model largely ignored text. And 9% of listings had a photo that belonged to a different listing, because an upstream job matched on a key that got reused.

That second problem is the subject. Fusion is the modelling question: where do the modalities meet? Alignment is the data question: are these two things actually about the same thing? Fusion has three or four standard answers you can pick between in an afternoon. Alignment is months of work, and it decides whether any of the answers help.

Think of It Like This

Subtitles half a second out

You've watched a film with subtitles that ran slightly early. Not by much — half a second.

It's unbearable, and precisely because nothing is wrong. Every line of dialogue is translated correctly. Every frame is the right frame. The two streams are individually perfect and jointly useless, because you read the punchline while the character is still setting it up.

Nobody shipped that subtitle file thinking it was broken. It passed every check anyone ran: right number of lines, right language, right encoding.

That's a misaligned dataset. Two clean modalities, one offset, and a model that learns the offset instead of the meaning.

How It Actually Works

Three places to fuse

Early fusion concatenates the inputs, or lightly processed versions of them, and hands one vector to one model. Cheap to write. Two problems: whichever modality has more dimensions or larger values dominates unless you're careful with scaling, and a missing modality changes the input shape, so there's no graceful degradation.

Late fusion runs one model per modality and combines the predictions — average, weighted vote, or a small model over the outputs. Robust: a missing modality just drops out of the vote. And it can't learn any interaction, because the modalities never meet before the answer. If the useful signal is "the photo shows a red dress and the title says blue", late fusion cannot see it.

Joint or intermediate fusion gives each modality its own encoder, projects both into a shared space, and lets them interact there — concatenation of the projections, or cross-attention where one modality attends over the other. This is what works, and it's what every current multimodal model does.

Alignment is three separate problems

Instance alignment: this image goes with this text. Sounds trivial until you're joining two systems on a key that one of them recycles.

Temporal alignment: this video frame, this audio window and this caption refer to the same instant. Sub-second offsets are invisible to inspection and destructive to training.

Semantic alignment: the two modalities end up in one space where distance means the same thing regardless of which encoder produced the vector. That's what a contrastive objective builds — pull matched pairs together, push mismatched pairs apart — and it's the mechanism behind searching images with a text query.

The data problems that decide the project

Pair quality. Web alt-text is noisy, so a large fraction of image-caption pairs are weakly related or unrelated. Filtering pairs by a similarity threshold from a smaller clean model reliably beats scaling the architecture, for a while. Data quality filtering is the same discipline applied here.

Scale asymmetry. Millions of images and thousands of captions is normal. The rare modality is the bottleneck, and padding it with generated text imports whatever the generator believes.

Missing modalities at serving. This is the normal case, not the exception — a listing with no photo, a call with no transcript yet, a patient record missing the scan. Decide the strategy at training time: modality dropout (randomly drop a modality during training so the model learns to cope), a learned missing-modality token, or a late-fusion fallback path.

One modality dominating. If text alone answers 90% of your benchmark, the model will learn to read text and ignore pixels, and your evaluation will call that success. Test with modalities ablated: score text-only, image-only, and both. If both is no better than the best single one, you have not built a multimodal model.

Worked example

Two images and their two captions, each encoder's output projected to the same width and normalised to unit length. Image vectors $[1, 0]$ and $[0, 1]$ ; caption vectors $[0.8, 0.6]$ and $[0.6, 0.8]$ .

Because everything is unit length, a dot product is cosine similarity. The 2 × 2 similarity matrix comes out as $0.8$ on the diagonal and $0.6$ off it. Matched pairs score higher than mismatched ones by 0.2, and the contrastive objective's whole job is widening that gap.

Now shift the caption file by one row. Image one is paired with caption two, so the pairs the loss treats as correct are the $0.6$ entries, and it spends the entire training run pushing the $0.8$ entries — the genuinely matched ones — apart. Nothing errors. The loss goes down. You have trained the model to believe the wrong pairing.

Show Me the Code

1import numpy as np2
3img: np.ndarray = np.array([[1.0, 0.0], [0.0, 1.0]])   # two images, projected and unit length4txt: np.ndarray = np.array([[0.8, 0.6], [0.6, 0.8]])   # their captions, same shared space5
6def similarity(a: np.ndarray, b: np.ndarray) -> np.ndarray:7    """Unit vectors, so a dot product is already the cosine."""8    return a @ b.T9
10S: np.ndarray = similarity(img, txt)11matched: np.ndarray = np.diag(S)12mismatched: np.ndarray = np.diag(np.fliplr(S))         # what an off-by-one pairing would reward13
14print(np.round(S, 2).tolist())                          # -> [[0.8, 0.6], [0.6, 0.8]]15print(np.round(matched, 2).tolist())                    # -> [0.8, 0.8]16print(np.round(mismatched, 2).tolist())                 # -> [0.6, 0.6]17print(round(float(matched.mean() - mismatched.mean()), 2))   # -> 0.2  the gap training widens18print(round(float(np.linalg.norm(txt[0])), 4))          # -> 1.0  the vectors really are unit length

Swap matched and mismatched and you have the off-by-one bug: same code, same loss curve, opposite lesson.

Watch Out For

A fixed offset between two modalities

A caption track that starts 400ms early. Frames indexed from 1 and audio windows indexed from 0. A labelling tool that timestamps a click when the annotator pressed the key rather than when the event appeared.

Every one of these produces a dataset where each modality is individually correct and the pairing is systematically wrong. There's no null, no shape mismatch, no exception. Training proceeds, the loss falls, and the model learns a blurred average because the signal it's being shown is genuinely inconsistent.

It's also nearly invisible in review. Spot-check ten samples and a 400ms offset on a two-second window looks like a normal amount of imprecision.

Check it with a measurement rather than an eyeball. Take a subset, deliberately shift one modality by a range of offsets, and score alignment at each. If the best score is at −400ms rather than 0, you've found the bug and its size. Do that once per data source, and re-do it whenever an upstream tool version changes.

Requiring every modality when production has one

The model takes an image and a description. It was trained on listings that had both, because that's what the export contained.

In production, 30% of new listings have no photo for the first few hours. The serving code fills the image slot with zeros, since the tensor has to be the right shape. A zero image isn't a missing image — it's a specific black rectangle, a valid input the model has an opinion about, and it scores those listings confidently and wrongly.

Nothing in the metrics catches it, because your test set came from the same complete-records export.

Decide this before training. Use modality dropout so the model sees missing inputs during training and learns not to rely on them, or add a learned "absent" token per modality, or keep a late-fusion fallback for the single-modality case. Then build an evaluation slice per availability pattern — both, image only, text only — and report all three.

The Quick Version

Fusion is the modelling question, alignment is the data question, and alignment is where the work and the failures are.
Early fusion concatenates inputs, so the wider modality dominates and a missing one breaks the shape.
Late fusion combines predictions, survives a missing modality, and can never learn a cross-modal interaction.
Joint fusion gives each modality an encoder into a shared space where they interact. That's what works.
Alignment is three problems: instance, temporal, and semantic. A contrastive objective builds the semantic one.
Filtering noisy pairs beats scaling the model, for a good while.
Missing modalities at serving are the normal case. Plan for them with modality dropout, an absent token, or a fallback path.
If one modality answers the benchmark alone, the model will ignore the other and your evaluation will call it success. Score with modalities ablated.
Matched pairs scored 0.8 and mismatched 0.6. An off-by-one pairing spends training pushing the right pairs apart.