Layer 6: Inference¶

The Inference layer is the capstone of ZigLlama. It composes every layer beneath it -- tensors, linear algebra, neural primitives, transformer blocks, and model loading -- into a complete text generation pipeline. This is where a static collection of weight matrices becomes a system that produces language.

Given a prompt and a set of configuration parameters, the inference layer autoregressively generates text one token at a time: running a forward pass through the model, converting raw logits into a probability distribution, selecting the next token via a sampling strategy, and repeating until a stop condition is met. The remaining modules in this layer address the engineering challenges that arise when you want this loop to be fast, controllable, and production-ready.

Learning Objectives¶

After completing the eight modules in this layer you will be able to:

1. Implement autoregressive text generation from first principles, including the forward pass, logit processing, and token selection loop.
2. Compare sampling strategies -- greedy, temperature, top-k, top-p, and combined -- and predict their effects on output quality and diversity.
3. Explain advanced sampling methods (Mirostat, typical, tail-free, contrastive search) and select appropriate strategies for different tasks.
4. Design a KV cache that reduces per-token attention cost from \( O(n^2 d) \) to \( O(n \cdot d) \), including sliding-window variants.
5. Build a streaming generation pipeline with thread-safe token buffers, callback-based delivery, and natural break detection.
6. Architect a batch processing system with dynamic batching, request queuing, and throughput scaling analysis.
7. Constrain generation output using grammar specifications (JSON, regex, context-free grammars) via token masking.
8. Profile inference performance using RAII measurement blocks, percentile statistics, and the roofline model for bottleneck identification.

Prerequisites¶

Components Overview¶

Inference Pipeline Architecture¶

The following diagram shows the complete data flow from a user prompt to generated text. Every box corresponds to a module in this layer.

flowchart TD
    PROMPT["User Prompt"]

    subgraph "Layer 6 -- Inference"
        TOK["Tokenize Prompt"]
        FWD["Forward Pass (Layer 5 Model)"]
        LOGITS["Raw Logits"]
        REP["Repetition Penalty"]
        GRAM["Grammar Mask (optional)"]
        SAMP["Sampling Strategy"]
        ADV["Advanced Sampling (optional)"]
        CACHE["KV Cache Update"]
        CHECK["Stop Condition Check"]
        BUF["Stream Buffer"]
        BATCH["Batch Scheduler"]
    end

    OUTPUT["Generated Text"]

    PROMPT --> TOK --> FWD
    FWD --> LOGITS --> REP --> GRAM --> SAMP
    SAMP --> ADV
    ADV --> CACHE --> CHECK
    CHECK -->|continue| FWD
    CHECK -->|stop| OUTPUT
    CHECK -->|stream| BUF --> OUTPUT
    PROMPT --> BATCH --> TOK

    style SAMP fill:#d5f5e3,stroke:#1e8449
    style CACHE fill:#d6eaf8,stroke:#2e86c1
    style BUF fill:#fdebd0,stroke:#e67e22
    style GRAM fill:#e8daef,stroke:#7d3c98

Dependency Graph¶

Within the Inference layer, the modules depend on each other as follows:

graph LR
    GEN["generation.zig"]
    ADV["advanced_sampling.zig"]
    KV["kv_cache.zig"]
    STR["streaming.zig"]
    BAT["batching.zig"]
    GRM["grammar_constraints.zig"]
    PRF["profiling.zig"]

    STR --> GEN
    BAT --> GEN
    BAT --> KV
    PRF --> GEN
    PRF --> BAT
    PRF --> KV
    ADV --> GEN
    GRM --> GEN

generation.zig is the central module; every other inference module depends on it either directly or through the types it exports.

Suggested Reading Order¶

1. Text Generation -- start with the core autoregressive loop and generation configuration.
2. Sampling Strategies -- understand how tokens are selected from the model's probability distribution.
3. Advanced Sampling -- explore entropy-targeting and information-theoretic sampling methods.
4. KV Cache -- learn the key optimisation that makes autoregressive generation practical.
5. Streaming -- see how tokens are delivered in real time to users.
6. Batch Processing -- scale throughput with dynamic batching and request scheduling.
7. Grammar Constraints -- constrain output to valid JSON, regex patterns, or formal grammars.
8. Profiling -- measure, benchmark, and optimise the entire pipeline.

Key Design Decisions¶

Performance Summary¶

References¶