Performance Analysis¶

This page presents the performance characteristics of ZigLlama's Layer 2 kernels -- matrix multiplication and quantized arithmetic -- using systematic benchmarking, roofline modeling, and memory-bandwidth analysis. The goal is not just to report numbers but to build an analytical framework for predicting performance and identifying bottlenecks.

1. Benchmarking Methodology¶

Measurement Protocol¶

Statistical Reporting¶

For each benchmark, we report the following statistics:

Environment Control¶

pub const BenchConfig = struct {
    warmup_iterations: usize = 10,
    measurement_iterations: usize = 100,
    /// Pin to performance cores (Linux: isolcpus, taskset).
    pin_to_core: ?usize = null,
    /// Disable turbo boost for stable frequency.
    disable_turbo: bool = true,

    pub fn run(self: BenchConfig, comptime kernel: anytype, args: anytype) BenchResult {
        // Warmup phase
        for (0..self.warmup_iterations) |_| {
            @call(.auto, kernel, args);
        }

        // Measurement phase
        var times: [self.measurement_iterations]u64 = undefined;
        for (0..self.measurement_iterations) |i| {
            const start = std.time.nanoTimestamp();
            @call(.auto, kernel, args);
            const end = std.time.nanoTimestamp();
            times[i] = @intCast(end - start);
        }

        return BenchResult.compute(&times);
    }
};

2. Matrix Multiplication Performance¶

Results by Matrix Size¶

All measurements on Intel i7-13700K (Performance cores, 5.4 GHz turbo, 32 KiB L1d, 1.25 MiB L2, 30 MiB L3). Kernel: matmulSIMD_f32_blocked with AVX2+FMA.

Performance Scaling¶

xychart-beta
    title "Matrix Multiplication: GFLOPS by ISA"
    x-axis ["64", "256", "512", "1024", "4096"]
    y-axis "GFLOPS" 0 --> 70
    bar "Scalar" [10.5, 2.8, 2.7, 2.6, 2.6]
    bar "SSE" [28.4, 8.8, 9.4, 9.8, 10.2]
    bar "AVX" [42.1, 16.0, 17.8, 18.9, 19.7]
    bar "AVX2+FMA" [65.5, 24.0, 27.5, 31.2, 33.8]

3. SIMD Speedup Analysis¶

Speedup Relative to Scalar¶

Speedup Convergence¶

xychart-beta
    title "SIMD Speedup vs Matrix Size (relative to scalar)"
    x-axis ["64", "256", "512", "1024", "4096"]
    y-axis "Speedup" 0 --> 14
    line "SSE (theoretical 4x)" [2.7, 3.1, 3.5, 3.8, 3.9]
    line "AVX (theoretical 8x)" [4.0, 5.7, 6.6, 7.3, 7.6]
    line "AVX2+FMA" [6.2, 8.6, 10.2, 12.0, 13.0]

The speedup curves approach asymptotic values as matrix size increases: - SSE converges near 4x (limited by 4 lanes, no FMA). - AVX converges near 8x. - AVX2+FMA exceeds 8x due to the FMA throughput doubling.

4. Roofline Model¶

Operational Intensity of Matrix Multiplication¶

For \( C = A \times B \) with \( A \in \mathbb{R}^{M \times K} \), \( B \in \mathbb{R}^{K \times N} \):

• FLOPs: \( 2MNK \) (one multiply + one add per inner product element)
• Bytes transferred: \( 4(MK + KN + MN) \) for f32

For square matrices (\( M = N = K = n \)):

Roofline Diagram¶

xychart-beta
    title "Roofline Model: Performance vs Operational Intensity"
    x-axis "Operational Intensity (FLOP/byte)" [0.1, 0.5, 1, 2, 5, 10, 50, 100]
    y-axis "GFLOPS" 0 --> 100
    line "Bandwidth ceiling (50 GB/s)" [5, 25, 50, 86.4, 86.4, 86.4, 86.4, 86.4]
    line "Compute ceiling" [86.4, 86.4, 86.4, 86.4, 86.4, 86.4, 86.4, 86.4]

5. Quantization Throughput¶

Dequantization Kernel Performance¶

Dequantization is the critical path for quantized inference: every quantized matrix multiplication requires on-the-fly dequantization of weight blocks.

Combined Quantized Matmul¶

The end-to-end performance of quantized matrix multiplication (dequant + matmul) for a typical transformer weight matrix (\( 4096 \times 4096 \)):

6. Memory Bandwidth Analysis¶

When is Matmul Compute-Bound vs Memory-Bound?¶

For inference with quantized weights, the matmul is a matrix-vector product (batch size 1), not a matrix-matrix product. The operational intensity drops dramatically:

where the numerator is FLOPs per row (\( 2n \) for dot product of length \( n \)) and the denominator is bytes per row (\( n \cdot b/8 \) for quantized weights + \( 4n \) for f32 activations... simplified to per-element).

For a single row with \( n = 4096 \) and Q4_K (\( b = 4.5 \)):

This is above the ridge point (1.73), so single-token quantized matmul is moderately compute-bound -- but only barely. It sits near the ridge of the roofline.

Bandwidth Utilization¶

Bandwidth Saturation by Format¶

xychart-beta
    title "Memory Bandwidth Used During Matmul (4096x4096, batch=1)"
    x-axis ["F32", "F16", "Q8_0", "Q4_K", "IQ4_NL", "IQ2_XS"]
    y-axis "Bandwidth (GB/s)" 0 --> 50
    bar [17.0, 24.1, 35.2, 41.8, 42.3, 44.7]

Higher-compression formats push closer to the memory bandwidth ceiling, extracting maximum value from available bandwidth.

7. Optimization Recommendations¶

Summary of Bottlenecks¶

Actionable Recommendations¶

Performance Comparison with llama.cpp¶

For context, the following table shows ZigLlama's Layer 2 kernel performance relative to llama.cpp on the same hardware:

Appendix: Benchmark Reproduction¶

To reproduce the benchmarks in this page:

// Run with: zig build bench -- --layer2
const bench = @import("bench");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();

    const config = BenchConfig{
        .warmup_iterations = 10,
        .measurement_iterations = 100,
        .disable_turbo = true,
    };

    // Matrix multiplication benchmarks
    for ([_]usize{ 64, 256, 512, 1024, 4096 }) |n| {
        const A = try gpa.allocator().alloc(f32, n * n);
        defer gpa.allocator().free(A);
        const B = try gpa.allocator().alloc(f32, n * n);
        defer gpa.allocator().free(B);
        const C = try gpa.allocator().alloc(f32, n * n);
        defer gpa.allocator().free(C);

        // Fill with random data
        fillRandom(A);
        fillRandom(B);

        const result = config.run(matmulSIMD_f32_blocked, .{ A, B, C, n, n, n });
        const gflops = @as(f64, 2.0 * n * n * n) / result.mean_ns;
        std.debug.print("{}x{}: {d:.1} GFLOPS (mean={d:.1}ms, p95={d:.1}ms)\n",
            .{ n, n, gflops, result.mean_ns / 1e6, result.p95_ns / 1e6 });
    }
}

References¶