Multi-Modal Vision-Language Models¶

Multi-modal models extend language models to process both text and images within a single architecture. Rather than building entirely new models, the dominant approach connects a pre-trained vision encoder (typically a Vision Transformer, ViT) to a pre-trained language model through a learned projection layer. This modular design allows each component to be trained separately and then aligned, reducing the enormous cost of training a multi-modal system from scratch¹.

1. Architecture Overview¶

The high-level pipeline is:

1. Split the image into fixed-size patches
2. Encode patches with a Vision Transformer to get visual features
3. Project visual features into the language model's embedding space
4. Concatenate visual tokens with text tokens
5. Process the combined sequence with the language model

2. Key Innovations¶

2.1 Vision Transformer (ViT)¶

The Vision Transformer (Dosovitskiy et al., 2021)² treats an image as a sequence of patches, analogous to how a language model treats text as a sequence of tokens:

The patch embeddings are then processed by a standard transformer encoder (bidirectional attention, like BERT).

2.2 Cross-Modal Projection¶

The vision encoder produces features in its own embedding space (\( d_v \)-dimensional). The language model operates in a different space (\( d_l \)-dimensional). A projection layer bridges the two:

2.3 Sequence Construction¶

After projection, the visual tokens are treated identically to text tokens:

The language model processes this combined sequence autoregressively. The visual tokens provide context; the model generates text tokens conditioned on both the image and the text prompt.

3. Architecture Diagram¶

flowchart LR
    subgraph VISION["Vision Encoder (ViT)"]
        IMG["Input Image\n(H x W x C)"] --> PATCH["Patch Extraction\n(P x P patches)"]
        PATCH --> PROJ_V["Patch Projection\n+ Position Embedding"]
        PROJ_V --> VIT_ENC["ViT Transformer\nEncoder (L layers)"]
        VIT_ENC --> VIS_FEAT["Visual Features\n[N, d_vision]"]
    end

    subgraph BRIDGE["Cross-Modal Projection"]
        VIS_FEAT --> MLP_PROJ["MLP Projection\nd_vision -> d_language"]
        MLP_PROJ --> VIS_TOKENS["Visual Tokens\n[N, d_language]"]
    end

    subgraph LANGUAGE["Language Model (Decoder)"]
        TEXT["Text Tokens"] --> TEXT_EMB["Text Embedding"]
        VIS_TOKENS --> CONCAT["Concatenate\n[visual ; text]"]
        TEXT_EMB --> CONCAT
        CONCAT --> LLM["LLM Decoder\n(LLaMA / Gemma / etc.)"]
        LLM --> OUTPUT["Generated Text"]
    end

4. Configuration Parameters¶

4.1 Vision Encoder¶

4.2 Multi-Modal Configuration¶

5. Mathematical Formulation¶

5.1 Patch Embedding¶

For image \( I \in \mathbb{R}^{H \times W \times 3} \) with patch size \( P \):

5.2 Vision Transformer Forward Pass¶

For each layer \( l \):

[ Z'^{(l)} = Z^{(l)} + \text{MHA}(\text{LN}(Z^{(l)})) ] [ Z^{(l+1)} = Z'^{(l)} + \text{FFN}(\text{LN}(Z'^{(l)})) ]

5.3 MLP Projection (LLaVA-style)¶

where \( W_1 \in \mathbb{R}^{d_v \times d_h} \), \( W_2 \in \mathbb{R}^{d_h \times d_l} \), and typically \( d_h = d_l \).

5.4 Combined Sequence Processing¶

The language model attends over both visual and text tokens when generating.

5.5 Image Preprocessing¶

6. Zig Implementation¶

6.1 Configuration Structs¶

pub const VisionConfig = struct {
    image_size: u32 = 336,
    patch_size: u32 = 14,
    d_vision: u32 = 1024,
    n_layers: u32 = 24,
    n_heads: u32 = 16,
    d_ff: u32 = 4096,
    norm_eps: f32 = 1e-6,
    activation: ActivationType = .quick_gelu,

    pub fn numPatches(self: VisionConfig) u32 {
        const grid = self.image_size / self.patch_size;
        return grid * grid;
    }
};

pub const ProjectionConfig = struct {
    d_vision: u32,
    d_language: u32,
    d_hidden: ?u32 = null,    // null = use d_language
    projection_type: enum { linear, mlp, cross_attention } = .mlp,
};

pub const MultiModalConfig = struct {
    vision: VisionConfig,
    projection: ProjectionConfig,
    language_model: anytype,     // LLaMA, Gemma, etc.
    max_visual_tokens: u32 = 576,
};

6.2 Patch Embedding¶

pub const PatchEmbedding = struct {
    proj: Linear,             // [patch_size^2 * 3, d_vision]
    cls_token: Tensor(f32),   // [1, d_vision]
    pos_embed: Tensor(f32),   // [1 + n_patches, d_vision]

    pub fn forward(self: *PatchEmbedding, image: Tensor(f32)) !Tensor(f32) {
        const patches = extractPatches(image, self.patch_size);
        var embeddings = try self.proj.forward(patches);

        // Prepend [CLS] token
        var with_cls = try prependToken(self.cls_token, embeddings);

        // Add positional embeddings
        return elementWiseAdd(with_cls, self.pos_embed);
    }
};

6.3 Vision-Language Projection¶

pub const VisionLanguageProjection = struct {
    w1: Linear,     // [d_vision, d_hidden]
    w2: Linear,     // [d_hidden, d_language]

    pub fn forward(self: *VisionLanguageProjection, vis_features: Tensor(f32)) !Tensor(f32) {
        // 2-layer MLP: GELU(x @ W1 + b1) @ W2 + b2
        const hidden = gelu(try self.w1.forward(vis_features));
        return self.w2.forward(hidden);
    }
};

6.4 Multi-Modal Forward Pass¶

pub const MultiModalModel = struct {
    vision_encoder: VisionTransformer,
    projection: VisionLanguageProjection,
    language_model: LLaMAModel,
    config: MultiModalConfig,

    pub fn forward(
        self: *MultiModalModel,
        image: ?Tensor(f32),      // null for text-only
        text_tokens: []const u32,
    ) ![]f32 {
        var combined_embeds: Tensor(f32) = undefined;

        if (image) |img| {
            // 1. Encode image through ViT
            const vis_features = try self.vision_encoder.forward(img);

            // 2. Project to language embedding space
            const vis_tokens = try self.projection.forward(vis_features);

            // 3. Get text embeddings
            const text_embeds = self.language_model.embedding.lookup(text_tokens);

            // 4. Concatenate: [visual_tokens, text_tokens]
            combined_embeds = try concatenate(vis_tokens, text_embeds);
        } else {
            combined_embeds = self.language_model.embedding.lookup(text_tokens);
        }

        // 5. Process through language model decoder
        return self.language_model.forwardEmbeddings(combined_embeds);
    }
};

7. Variants¶

8. Educational Value¶

9. References¶