""" DeepSeek-V4 Architecture Analysis ================================== Detailed technical comparison between DeepSeek-V3 and DeepSeek-V4, documenting all architectural innovations from the V4 technical report. """ ARCHITECTURE_SPEC = { "DeepSeek-V4-Pro": { # Core "total_params": "1.6T", "active_params": "49B", "hidden_size": 7168, "num_layers": 61, "num_heads": 128, "vocab_size": 129280, "context_length": "1M tokens", # MLA "q_lora_rank": 1536, "head_dim": 512, # unified (V3 had separate kv_lora_rank=512 + nope=128 + rope=64 + v=128) "qk_rope_head_dim": 64, "o_groups": 16, "o_lora_rank": 1024, # Compressed Sparse Attention "window_size": 128, "csa_ratio": 4, # 4:1 compression with overlapping windows "hca_ratio": 128, # 128:1 heavy compression "index_n_heads": 64, "index_head_dim": 128, "index_topk": 1024, "has_attention_sink": True, # MoE "n_routed_experts": 384, "n_shared_experts": 1, "num_experts_per_tok": 6, "moe_intermediate_size": 3072, "scoring_func": "sqrtsoftplus", "route_scale": 2.5, "num_hash_layers": 3, "swiglu_limit": 10.0, "expert_dtype": "fp4", # Hyper-Connections "hc_mult": 4, "hc_sinkhorn_iters": 20, # MTP "num_mtp_layers": 1, # Training "optimizer": "Muon", "training_tokens": "32T+", "precision": "FP8 (linear) + FP4 (experts)", # RoPE "rope_theta": 10000, "rope_scaling": "YaRN (factor=16, original=65536)", "compress_rope_theta": 160000, }, "DeepSeek-V4-Flash": { "total_params": "284B", "active_params": "13B", "hidden_size": 4096, # estimated "num_layers": 61, # estimated same depth "num_heads": 64, "vocab_size": 129280, "context_length": "1M tokens", "n_routed_experts": 128, # estimated }, } INNOVATIONS = """ ================================================================================ DeepSeek-V4 Architectural Innovations ================================================================================ 1. COMPRESSED SPARSE ATTENTION (CSA) V3: Standard MLA with full KV computation (O(L²) for L tokens) V4: Hybrid sliding window + learned KV compression - Sliding window: Each token attends to nearest 128 tokens (local context) - CSA (4:1): Learned gated pooling compresses 4 tokens → 1, with overlapping windows for smoother boundaries. The compressor uses: * Learned projection (wkv) + gating (wgate) * Absolute position embeddings (ape) within compression windows * RMSNorm on compressed output * RoPE on rope dimensions of compressed KV - HCA (128:1): Heavy compression for global context — 128 tokens → 1 - Indexer: Learned scoring mechanism selects top-k compressed positions * Has its own dedicated KV compressor with Hadamard rotation * Uses dot-product attention + per-head importance weights * Trained with QAT (Quantization-Aware Training) - Attention sink: Learnable per-head bias (scalar) that acts as a "catch-all" for information that doesn't belong to any specific KV position Result: Only 27% of single-token inference FLOPs and 10% of KV cache vs V3.2 2. UNIFIED HEAD DIMENSION V3: Separate dimensions — kv_lora_rank=512, qk_nope_head_dim=128, qk_rope_head_dim=64, v_head_dim=128 V4: Single head_dim=512 that encompasses all roles The key and value share the same head_dim=512 representation, with the last qk_rope_head_dim=64 dimensions reserved for RoPE (applied separately). 3. GROUPED LOW-RANK OUTPUT PROJECTION V3: Standard wo: [num_heads * v_head_dim, hidden_size] V4: Two-stage with grouping: wo_a: heads split into o_groups=16 groups, each compressed to o_lora_rank=1024 wo_b: [o_groups * o_lora_rank, hidden_size] This reduces output projection parameters while maintaining expressivity through group-level compression. 4. MANIFOLD-CONSTRAINED HYPER-CONNECTIONS (mHC) V3: Standard residual connections: x = x + sublayer(x) V4: Maintains hc_mult=4 copies of hidden state throughout the network Pre-connection: - Input: [batch, seq, 4, hidden_size] - Compute mixing coefficients via linear projection + RMSNorm scaling - Split into: pre_weights (sigmoid), post_weights (2*sigmoid), combination (Sinkhorn) - Reduce: weighted sum of 4 copies → 1 for sublayer input Post-connection: - Expand sublayer output: 1 → 4 copies via post_weights - Mix with residual via doubly-stochastic combination matrix (Sinkhorn normalized) The Sinkhorn normalization (20 iterations) ensures the combination matrix is doubly-stochastic, constraining the mixing to a manifold that preserves information flow stability across 61 layers. At the model output, a simpler sigmoid-gated HC reduces 4 → 1 for the final hidden state. 5. HASH ROUTING V3: All layers use score-based routing V4: First 3 layers use hash routing (deterministic expert assignment by token ID) Hash routing eliminates load balancing issues for early layers where token representations are still close to embeddings. The hash table (tid2eid) maps each token in the vocabulary to a fixed set of experts. 6. SQRTSOFTPLUS SCORING V3: sigmoid scoring for expert routing V4: sqrtsoftplus = sqrt(softplus(x)) = sqrt(log(1 + exp(x))) Properties: - Smoother gradient behavior than sigmoid near saturation - Always positive (no dead experts from zero gates) - Sub-linear growth prevents extreme weight concentration 7. SWIGLU CLAMPING V3: Standard SwiGLU: silu(gate) * up V4: Clamped SwiGLU with swiglu_limit=10.0: up = clamp(up, -10, 10) gate = clamp(gate, max=10) output = silu(gate) * up Prevents numerical overflow in FP4/FP8 expert computation. 8. FP4 EXPERT WEIGHTS V3: FP8 block-wise quantized expert weights V4: FP4 (float4_e2m1fn) expert weights with E8M0 per-32-element scales Further 2x compression of expert parameters. The MoE accounts for the vast majority of parameters (384 experts × 3 FFN matrices each), so FP4 experts dramatically reduce model size (1.6T total but manageable storage). 9. MUON OPTIMIZER V3: AdamW (β1=0.9, β2=0.95) V4: Muon optimizer Muon uses momentum-based, matrix-valued updates for faster convergence. The paper reports improved training stability across 32T+ tokens. 10. MULTI-TOKEN PREDICTION (MTP) V3: MTP with depth=1 V4: MTP with depth=1 (similar to V3) The MTP block: - Takes hidden state from last layer (in HC format: [b, s, 4, d]) - Combines with next-token embedding via separate projections - Processes through a full transformer block - Reduces HC copies and projects to vocabulary for auxiliary prediction loss - Shares embedding and LM head weights with main model ================================================================================ Compression Pattern (compress_ratios for V4-Pro, 61 layers): ================================================================================ Layer 0: 128 (HCA only — heavy global compression) Layer 1: 128 (HCA) Layer 2: 4 (CSA — fine-grained compression with indexer) Layer 3: 128 (HCA) Layer 4: 4 (CSA) ... alternating 128/4 pattern Layer 60: 0 (no compression — pure sliding window) This alternating pattern ensures every pair of layers provides both: - Fine-grained local+compressed attention (CSA, ratio=4) - Broad global context with heavy compression (HCA, ratio=128) The last layer (60) has no compression (ratio=0), using only sliding window, which suggests it focuses purely on local token interactions for final prediction. """ if __name__ == "__main__": print(INNOVATIONS) print("\n\nV4-Pro Full Specification:") for k, v in ARCHITECTURE_SPEC["DeepSeek-V4-Pro"].items(): print(f" {k}: {v}")