Phi¶

The Phi family represents Microsoft Research's sustained investigation into the hypothesis that high-quality training data can compensate for reduced parameter counts. Beginning with Phi-1 (2023) and continuing through Phi-3 (2024), each generation demonstrated that small models trained on carefully curated "textbook-quality" data can match or exceed models several times their size on standard benchmarks.

1. Architecture Overview¶

Phi-1 targeted code generation exclusively. Phi-2 broadened to general language understanding while retaining the "small but capable" philosophy. Phi-3 introduced long-context variants (up to 128K tokens) and a Mixture of Experts variant (PhiMoE).

2. Key Innovations¶

2.1 Partial Rotary Embeddings¶

Unlike LLaMA, which applies Rotary Position Embeddings (RoPE) to the full head dimension, Phi applies RoPE to only a fraction of the query and key dimensions. The remaining dimensions receive no positional encoding.

2.2 QK LayerNorm¶

Phi-2 and later variants apply LayerNorm to the query and key projections before computing attention scores. This stabilizes the magnitude of the dot products, reducing the need for careful learning-rate tuning.

2.3 Data-Centric Training¶

The Phi series was among the first to demonstrate that a 1.3B model trained on "textbook-quality" synthetic data could outperform 10B+ models trained on raw web crawls. This is an architectural-agnostic insight but deeply influenced the model's design constraints.

3. Architecture Diagram¶

flowchart TD
    INPUT["Token IDs"] --> EMB["Token Embedding"]
    EMB --> BLOCKS

    subgraph BLOCKS["N x PhiDecoderLayer"]
        LN1["LayerNorm"] --> ATTN
        subgraph ATTN["Phi Attention"]
            QKV["QKV Projection"] --> QKN["QK LayerNorm"]
            QKN --> PROPE["Partial RoPE\n(first d_rot dims only)"]
            PROPE --> DOT["Scaled Dot-Product"]
            DOT --> OUT_PROJ["Output Projection"]
        end
        ATTN --> RES1["Residual Add"]
        RES1 --> LN2["LayerNorm"]
        LN2 --> FFN["FFN (GELU / SwiGLU)"]
        FFN --> RES2["Residual Add"]
    end

    BLOCKS --> FINAL_LN["Final LayerNorm"]
    FINAL_LN --> LM_HEAD["LM Head"]

4. Configuration Parameters¶

5. Mathematical Formulation¶

5.1 Partial RoPE Application¶

For position \( m \) and head dimension index \( i \):

The rotation is applied only to the first \( d_{\text{rot}} \) dimensions of \( q \) and \( k \), leaving the remaining \( d_h - d_{\text{rot}} \) dimensions unchanged.

5.2 Attention with QK Norm¶

where \(\text{LN}\) denotes LayerNorm applied independently to Q and K.

5.3 Feed-Forward (Phi-3 SwiGLU Variant)¶

Earlier Phi models use standard GELU activation without gating:

6. Zig Implementation¶

6.1 PhiConfig¶

pub const PhiConfig = struct {
    n_layers: u32,
    d_model: u32,
    n_heads: u32,
    n_kv_heads: u32,
    d_ff: u32,
    vocab_size: u32,
    max_seq_len: u32,
    partial_rotary_factor: f32,     // fraction of head dim to rotate
    qk_layernorm: bool,            // apply LayerNorm to Q and K
    rope_theta: f32 = 10000.0,
    norm_eps: f32 = 1e-5,
    activation: ActivationType = .gelu,

    pub fn headDim(self: PhiConfig) u32 {
        return self.d_model / self.n_heads;
    }

    pub fn rotaryDim(self: PhiConfig) u32 {
        return @intFromFloat(@as(f32, @floatFromInt(self.headDim()))
            * self.partial_rotary_factor);
    }
};

6.2 Forward Pass Overview¶

pub fn forward(self: *PhiModel, tokens: []const u32, pos: u32) ![]f32 {
    var hidden = self.embedding.lookup(tokens);

    for (self.layers) |*layer| {
        const normed = layer.input_norm.forward(hidden);

        // QKV projection
        var q = layer.wq.forward(normed);
        var k = layer.wk.forward(normed);
        const v = layer.wv.forward(normed);

        // Optional QK LayerNorm
        if (self.config.qk_layernorm) {
            q = layer.q_norm.forward(q);
            k = layer.k_norm.forward(k);
        }

        // Partial RoPE -- rotate only first rotary_dim dimensions
        const rot_dim = self.config.rotaryDim();
        applyPartialRoPE(&q, &k, pos, rot_dim);

        const attn_out = layer.attention.forward(q, k, v);
        hidden = residualAdd(hidden, attn_out);

        const ff_normed = layer.post_norm.forward(hidden);
        const ff_out = layer.ffn.forward(ff_normed);
        hidden = residualAdd(hidden, ff_out);
    }

    return self.output_norm.forward(hidden);
}

7. Variants¶

8. Educational Value¶

9. References¶