Layer 5: Model Architectures¶

Overview¶

Layer 5 implements complete neural network architectures that compose the primitives from Layers 1--4 (linear algebra, tensors, neural primitives, transformer components) into end-to-end language models. ZigLLM ships 18 model implementations spanning the most influential architectures in modern NLP, covering an estimated 80% of real-world deployment scenarios despite representing only 19% of the GGUF specification's 94 registered architecture identifiers.

Architecture Taxonomy¶

The 18 supported architectures are organized into four categories based on their structural characteristics and intended use cases.

graph TB
    subgraph "Decoder-Only (Autoregressive)"
        direction TB
        L[LLaMA / LLaMA 2] --> M[Mistral]
        L --> F[Falcon]
        L --> Q[Qwen]
        L --> P[Phi]
        G2[GPT-2] --> GJ[GPT-J]
        G2 --> GN[GPT-NeoX]
        G2 --> BL[BLOOM]
        L --> SC[StarCoder]
    end

    subgraph "Encoder-Only"
        B[BERT]
    end

    subgraph "State-Space"
        MA[Mamba]
    end

    subgraph "Multi-Modal"
        MM[Multi-Modal]
    end

    subgraph "Specialized Components"
        MoE[Mixture of Experts]
        BLAS[BLAS Integration]
        GE[Gemma]
    end

    style L fill:#4a9eff,color:#fff
    style G2 fill:#4a9eff,color:#fff
    style B fill:#ff9f43,color:#fff
    style MA fill:#10ac84,color:#fff
    style MM fill:#ee5a24,color:#fff

Category Breakdown¶

Architecture Comparison¶

The following table summarizes the key design choices across all supported architectures.

Parameter Scale¶

The architectures span a wide range of parameter counts, from sub-billion to hundreds of billions.

Learning Path¶

We recommend the following progression through the model architectures, ordered by conceptual complexity.

Beginner¶

1. GPT-2 -- The classic decoder-only transformer. Learned position embeddings, GELU activation, post-LayerNorm. Start here to understand the baseline.
2. LLaMA -- Modern best practices: RoPE, SwiGLU, RMSNorm, pre-normalization. The canonical reference architecture for 2023--2025 era models.
3. BERT -- Bidirectional encoder. Contrasts with decoder-only design; essential for understanding embeddings and classification tasks.

Intermediate¶

1. Mistral -- Sliding window attention and grouped-query attention. Introduces memory-efficient attention patterns.
2. Falcon -- Multi-query attention and parallel attention+FFN blocks. Demonstrates alternative efficiency strategies.
3. Qwen -- Advanced RoPE scaling (NTK-Aware, YARN, Dynamic NTK). The most comprehensive position encoding implementation.

Advanced¶

1. Mamba -- State-space models. A fundamentally different approach with \( O(n) \) sequence complexity instead of \( O(n^2) \).
2. Mixture of Experts -- Sparse routing, expert parallelism, load balancing. Scales parameter count without proportional compute increase.
3. Multi-Modal -- Vision encoders, cross-attention fusion, modality alignment. Extends language models to process images.

Source Code Organization¶

All model implementations reside in src/models/:

src/models/
  config.zig            # Shared ModelConfig, ActivationType, NormalizationType
  tokenizer.zig         # Vocabulary, SimpleTokenizer, BPETokenizer
  chat_templates.zig    # ChatMessage, TemplateType, ChatTemplate (10 formats)
  gguf.zig              # GGUFReader, GGMLType, tensor loading
  llama.zig             # LLaMAModel, LLaMAConfig, LLaMATransformerLayer
  mistral.zig           # MistralModel, GroupedQueryAttention, SlidingWindow
  gpt2.zig              # GPT2Model, learned position embeddings
  falcon.zig            # FalconModel, parallel attention+FFN
  qwen.zig              # QwenModel, RopeScaling, Dynamic NTK
  phi.zig               # PhiModel
  gptj.zig              # GPTJModel
  gpt_neox.zig          # GPTNeoXModel
  bloom.zig             # BLOOMModel, ALiBi
  mamba.zig             # MambaModel, selective state spaces
  bert.zig              # BERTModel, bidirectional attention
  gemma.zig             # GemmaModel
  starcoder.zig         # StarCoderModel
  mixture_of_experts.zig # MoE routing and expert dispatch
  multi_modal.zig       # Vision-language fusion

Infrastructure Pages¶

Before diving into individual architectures, the following pages cover the shared infrastructure that all models depend on:

• Model Configuration -- ModelConfig struct, preset configurations, validation, parameter counting, and memory estimation.
• Tokenization -- Vocabulary management, SimpleTokenizer, BPE algorithm, batch encoding, and subword theory.
• Chat Templates -- Multi-turn conversation formatting for 10 model families (Llama2, ChatML, Mistral, etc.).
• GGUF Model Loading -- Binary format parsing, metadata extraction, quantized tensor loading, and memory-mapped I/O.

What Comes Next¶

After understanding individual architectures, Layer 6 (Inference) covers how these models are used in practice: KV-cache management, sampling strategies, batched generation, streaming output, and performance profiling.

References¶