The 6-Layer Progressive Architecture¶

ZigLlama decomposes the problem of running a large language model into six progressively more abstract layers. A reader can begin at Layer 1 and build a working mental model before advancing to Layer 2; by the time they reach Layer 6 they have seen every concept needed to generate text from a transformer.

1. Architectural Philosophy¶

Why progressive layers?¶

Language models are deep stacks of repeated operations, but they are typically presented either as opaque library calls or as a wall of linear-algebra code. Neither approach is good for learning.

ZigLlama's layered design offers a middle path:

The dependency invariant¶

Every module in layer \( i \) may depend only on modules in layers strictly below \( i \). This invariant is enforced by the Zig compiler (circular imports are a compile error) and by code review.

2. Layer Dependency Diagram¶

graph TB
    subgraph "Layer 6 -- Inference"
        gen[generation.zig]
        kv[kv_cache.zig]
        stream[streaming.zig]
        batch[batching.zig]
        prof[profiling.zig]
        adv_samp[advanced_sampling.zig]
        gram[grammar_constraints.zig]
    end

    subgraph "Layer 5 -- Models"
        llama[llama.zig]
        cfg[config.zig]
        tok[tokenizer.zig]
        gguf[gguf.zig]
        bert[bert.zig]
        gpt2[gpt2.zig]
        mistral[mistral.zig]
        falcon[falcon.zig]
        phi[phi.zig]
        bloom[bloom.zig]
        gemma[gemma.zig]
        qwen[qwen.zig]
        mamba[mamba.zig]
        gptj[gptj.zig]
        gptneox[gpt_neox.zig]
        starcoder[starcoder.zig]
        moe[mixture_of_experts.zig]
        mm[multi_modal.zig]
        chat[chat_templates.zig]
    end

    subgraph "Layer 4 -- Transformers"
        attn[attention.zig]
        ff[feed_forward.zig]
        tb[transformer_block.zig]
    end

    subgraph "Layer 3 -- Neural Primitives"
        act[activations.zig]
        norm[normalization.zig]
        emb[embeddings.zig]
    end

    subgraph "Layer 2 -- Linear Algebra"
        matops[matrix_ops.zig]
        quant[quantization.zig]
        kquant[k_quantization.zig]
        iquant[iq_quantization.zig]
    end

    subgraph "Layer 1 -- Foundation"
        tensor[tensor.zig]
        mmap[memory_mapping.zig]
        gguf_fmt[gguf_format.zig]
        blas[blas_integration.zig]
        thread[threading.zig]
    end

    %% Layer 6 -> 5
    gen --> llama
    gen --> tok
    stream --> gen
    batch --> llama & tok & gen & kv
    prof --> gen & batch & kv

    %% Layer 5 -> 4
    llama --> attn & ff & tb
    llama --> norm & emb

    %% Layer 4 -> 3 & 2
    attn --> matops & act & norm
    ff --> matops & act
    tb --> attn & ff & norm

    %% Layer 3 -> 1
    act --> tensor
    norm --> tensor
    emb --> tensor

    %% Layer 2 -> 1
    matops --> tensor
    quant --> tensor
    kquant --> tensor
    iquant --> tensor

    %% Foundation internal
    gguf_fmt --> tensor & quant
    mmap --> tensor
    blas --> tensor & thread

3. Layer-by-Layer Reference¶

Layer 1 -- Foundation¶

Key types exported:

pub const Tensor = foundation.tensor.Tensor;     // Tensor(f32), Tensor(i32), ...
pub const MemoryMap = memory_mapping.MemoryMap;
pub const GGUFReader = gguf_format.GGUFReader;
pub const BlasConfig = blas_integration.BlasConfig;
pub const ThreadPoolConfig = threading.ThreadPoolConfig;

Dependencies: std only.

Test count: 6 (tensor shape, indexing, fill, matmul, print, error cases).

Layer 2 -- Linear Algebra¶

Key mathematical operation -- blocked matrix multiply:

Blocked into \( b \times b \) tiles to fit in L1 cache:

Dependencies: Layer 1 (Tensor).

Test count: 5 (SIMD correctness, quantise/dequantise round-trip, block-format layout, edge cases).

Layer 3 -- Neural Primitives¶

Key mathematical definitions:

RMSNorm (Zhang & Sennrich, 2019):

SwiGLU (Shazeer, 2020):

Rotary Position Embedding:

Dependencies: Layer 1 (Tensor).

Test count: 9 (activation properties, normalisation invariants, embedding lookup, RoPE rotation).

Layer 4 -- Transformers¶

Scaled dot-product attention (Vaswani et al., 2017):

Multi-head decomposition:

where each head computes:

Dependencies: Layers 1--3 (Tensor, matrix_ops, activations, normalization).

Test count: 11 (attention output shape, causal mask, FFN variants, block residual connection, pre/post-norm equivalence).

Layer 5 -- Models¶

Supported architectures (18)¶

Supporting modules¶

Dependencies: Layers 1--4.

Test count: 45 (config presets, tokeniser encode/decode, GGUF header parsing, LLaMA forward pass, model size calculations).

Layer 6 -- Inference¶

tokens = tokenize(prompt)
for step in 0..max_tokens:
    logits = model.forward(tokens)
    next_token = sample(logits, config)
    tokens.append(next_token)
    if next_token == EOS:
        break
return detokenize(tokens)

Dependencies: Layers 1--5.

Test count: 47 (sampling distributions, KV cache append/evict, streaming delivery order, batch scheduling, profiler accuracy).

4. Data Flow Diagram¶

The following sequence diagram traces a single inference request from raw text to generated output.

sequenceDiagram
    participant User
    participant Gen as TextGenerator<br/>(Layer 6)
    participant Tok as Tokenizer<br/>(Layer 5)
    participant Model as LLaMAModel<br/>(Layer 5)
    participant TB as TransformerBlock<br/>(Layer 4)
    participant Attn as MultiHeadAttention<br/>(Layer 4)
    participant FFN as FeedForward<br/>(Layer 4)
    participant NP as Neural Primitives<br/>(Layer 3)
    participant LA as Linear Algebra<br/>(Layer 2)
    participant KV as KVCache<br/>(Layer 6)

    User->>Gen: generate("Once upon a time")
    Gen->>Tok: encode(prompt)
    Tok-->>Gen: token_ids[]

    loop For each generation step
        Gen->>Model: forward(token_ids)
        Model->>NP: TokenEmbedding.lookup(token_ids)
        NP-->>Model: embeddings [seq_len, d_model]

        loop For each transformer layer
            Model->>TB: forward(hidden_state)
            TB->>NP: RMSNorm(hidden_state)
            TB->>Attn: forward(Q, K, V)
            Attn->>LA: matmulSIMD(Q, K^T)
            LA-->>Attn: attention scores
            Attn->>NP: softmax(scores / sqrt(d_k))
            Attn->>LA: matmulSIMD(weights, V)
            LA-->>Attn: attention output
            Attn-->>TB: attended
            TB->>TB: residual + attended
            TB->>NP: RMSNorm(hidden_state)
            TB->>FFN: forward(normalized)
            FFN->>LA: matmulSIMD (gate, up, down)
            FFN->>NP: SwiGLU activation
            FFN-->>TB: ffn_output
            TB->>TB: residual + ffn_output
            TB-->>Model: layer_output
            Model->>KV: cache(K, V, layer_id)
        end

        Model->>NP: RMSNorm(final_hidden)
        Model->>LA: output_projection
        Model-->>Gen: logits [vocab_size]
        Gen->>Gen: sample(logits)
        Gen-->>User: next_token (streamed)
    end

5. Parameter and Memory Budget¶

The table below shows approximate parameter counts and memory requirements for LLaMA-family models at different quantisation levels.

Memory breakdown for LLaMA-7B (Q4_0)¶

Summary¶

The six-layer architecture transforms the inherently complex task of language model inference into a sequence of self-contained, testable, and documentable modules. Each layer adds exactly one level of abstraction:

1. Foundation -- data structures and system interfaces.
2. Linear Algebra -- fast numerics and compression.
3. Neural Primitives -- non-linearities and normalisation.
4. Transformers -- attention and feed-forward composition.
5. Models -- architecture definitions and weight loading.
6. Inference -- generation loop and production optimisations.

This separation is the single most important architectural decision in ZigLlama and the one most responsible for its educational effectiveness.