Fusion Strategies: Early, Late and In-Between

Where do the modalities meet?

Fusion is the point at which information from different modalities is combined. In early fusion you combine low-level features (or raw tokens) first and let one model learn from the joint input; it can capture fine interactions, such as which word refers to which region, but needs paired training data and more compute. In late fusion each modality has its own model and only their outputs (scores or decisions) are combined, for example by averaging; it is simple, robust when one modality is missing, and easy to build from existing models, but it cannot learn cross-modal interactions. Intermediate fusion combines learned representations in the middle of the network, for example with cross-attention layers; most modern vision-language models live here. The trade-off is always interaction power versus data, compute and modularity.

Early vs late fusion in a few lines

Early fusion concatenates features into one vector for a joint model. Late fusion averages decisions from two separate models.

import numpy as np

image_feat = np.array([0.2, 0.7, 0.1])        # from an image encoder
text_feat = np.array([0.5, 0.5])              # from a text encoder

early = np.concatenate([image_feat, text_feat])   # one joint vector
print("joint feature:", early, early.shape)

p_image = np.array([0.7, 0.3])                # image model: P(spam), P(ok)
p_text = np.array([0.4, 0.6])                 # text model:  P(spam), P(ok)
late = (p_image + p_text) / 2                 # average the decisions
print("late fusion:", late)

Output:

joint feature: [0.2 0.7 0.1 0.5 0.5] (5,)
late fusion: [0.55 0.45]

Selection panel vs group discussion

Late fusion is like three interviewers scoring a candidate separately and averaging the marks. Early fusion is like the panel discussing every answer together before scoring. The discussion catches more nuance, but it takes longer and needs everyone in the room.

Quick check: A product team has a good image classifier and a good text classifier, very little paired data, and inputs where the image is often missing. Which fusion is the sensible start?

• Late fusion of the two models' scores
• Early fusion of raw pixels and tokens
• Training a vision-language model from scratch
• Dropping the text model entirely