Perplexity Evaluation¶

Perplexity is the standard intrinsic metric for language models. ZigLlama's evaluation module (src/evaluation/perplexity.zig) provides a configurable evaluator, a sliding-window strategy for sequences that exceed the model's context length, and a benchmark suite for comparing quantisation levels.

What is Perplexity¶

Given a sequence of \(N\) tokens \(x_1, x_2, \ldots, x_N\), perplexity is defined as:

Intuitively, perplexity measures how "surprised" the model is by the data. Lower perplexity means the model assigns higher probability to the observed tokens.

Related metrics computed alongside perplexity:

PerplexityConfig¶

pub const PerplexityConfig = struct {
    max_sequence_length: usize = 2048,
    batch_size: usize = 1,
    sliding_window: usize = 512,
    window_overlap: usize = 64,
    use_log_probs: bool = true,
    temperature: f32 = 1.0,
    normalize_probs: bool = true,
    verbose: bool = false,
};

PerplexityResult¶

Every evaluation returns a PerplexityResult:

pub const PerplexityResult = struct {
    perplexity: f64,
    log_perplexity: f64,
    bits_per_char: f64,
    bits_per_token: f64,
    total_tokens: usize,
    total_chars: usize,
    token_log_probs: []f64,
    sequence_perplexities: []f64,
    evaluation_time_ms: u64,
    peak_memory_usage: usize,
};

The token_log_probs array stores the per-position log-probability, enabling fine-grained analysis (e.g., identifying passages where the model struggles). sequence_perplexities is populated when evaluating a multi-sequence dataset.

Sliding Window Evaluation¶

When a token sequence exceeds max_sequence_length, the evaluator switches to a sliding-window strategy:

flowchart LR
    subgraph Window 1
        T1[t1...t512]
    end
    subgraph Window 2
        T2[t449...t960]
    end
    subgraph Window 3
        T3[t897...t1408]
    end
    T1 -- "overlap 64" --> T2
    T2 -- "overlap 64" --> T3

1. The first window spans positions \([0, W)\) where \(W\) = sliding_window.
2. Each subsequent window advances by \(W - O\) positions (\(O\) = window_overlap).
3. Within each window, log-probabilities are computed via a full forward pass.
4. Tokens in the overlap region are discarded from the second window to avoid double-counting. Only positions in the non-overlapping suffix contribute to the aggregate.

The overlap ensures that every token has sufficient left-context for accurate prediction, even at window boundaries.

Interpreting Results¶

What is "good" perplexity?¶

Perplexity depends on the vocabulary, tokeniser, dataset, and model size. The following table gives rough baselines on WikiText-103 for common model sizes:

Degradation thresholds¶

Benchmarking Quantisation Quality¶

A typical quantisation-quality workflow:

const std = @import("std");
const perplexity = @import("evaluation/perplexity.zig");

// 1. Evaluate FP16 baseline
var config = perplexity.PerplexityConfig{ .verbose = true };
var evaluator = perplexity.PerplexityEvaluator.init(
    allocator, config, model_fp16, tokenizer,
);
const baseline = try evaluator.evaluateDataset(wiki_dataset);

// 2. Evaluate Q4_K_M
var evaluator_q4 = perplexity.PerplexityEvaluator.init(
    allocator, config, model_q4km, tokenizer,
);
const q4_result = try evaluator_q4.evaluateDataset(wiki_dataset);

// 3. Compare
const comparison = perplexity.PerplexityUtils.compareResults(
    baseline, q4_result,
);
std.log.info("Relative degradation: {d:.2f}%",
    .{comparison.relative_difference * 100});

The PerplexityComparison struct returned by compareResults reports:

Benchmark Suite¶

For systematic evaluation across multiple datasets, use BenchmarkSuite:

var suite = perplexity.BenchmarkSuite.init(allocator, evaluator);
defer suite.deinit();

// Add standard benchmarks
try suite.generateSyntheticDataset("synthetic-small", 100, 256);
try suite.generateSyntheticDataset("synthetic-large", 50, 1024);

// Run all benchmarks
var results = try suite.runBenchmark();
defer results.deinit(allocator);

results.printReport();
try results.saveToFile("benchmark_results.json", allocator);

The suite supports the following standard benchmark identifiers via StandardBenchmarks:

• WIKITEXT_103
• PENN_TREEBANK
• LAMBADA
• HELLASWAG
• SYNTHETIC_SMALL / SYNTHETIC_LARGE

Confidence Intervals¶

PerplexityUtils.calculateConfidenceInterval computes percentile-based confidence bounds over per-sequence perplexity values:

const ci = PerplexityUtils.calculateConfidenceInterval(
    result.sequence_perplexities, 0.95,
);
std.log.info("95% CI: [{d:.2f}, {d:.2f}]", .{ ci.lower, ci.upper });

This is particularly useful when comparing two quantisation levels: if their 95 % confidence intervals overlap, the difference is likely not meaningful.

Source Reference¶