---
license: mit
language:
- en
base_model:
- deepseek-ai/DeepSeek-V4-Pro
- deepseek-ai/DeepSeek-V4-Flash
- deepseek-ai/DeepSeek-V4-Pro-Base
- deepseek-ai/DeepSeek-V4-Flash-Base
pipeline_tag: text-generation
---
# Open DeepSeek-V4: Community Reproduction

An open-source, HuggingFace-compatible reproduction of **DeepSeek-V4** — a 1.6T parameter Mixture-of-Experts language model with 49B activated parameters and 1M token context length.

Based on the [DeepSeek-V4 Technical Report](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf) and official [inference code](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/tree/main/inference).

## 🏗️ Architecture Overview

DeepSeek-V4 introduces several key innovations over DeepSeek-V3:

### 1. Multi-head Latent Attention (MLA) with Compressed Sparse Attention
- **Low-rank KV compression**: Joint KV projection to `head_dim=512` (vs V3's separate `kv_lora_rank=512` + `qk_nope_head_dim=128` + `v_head_dim=128`)
- **Low-rank Q compression**: `q_lora_rank=1536` with RMSNorm bottleneck
- **Grouped low-rank O projection**: `o_groups=16`, `o_lora_rank=1024` — output is split into groups, each compressed independently
- **Compressed Sparse Attention (CSA)**: Sliding window (128 tokens) + learned KV compression with gated pooling (4:1 ratio with overlapping windows)
- **Heavily Compressed Attention (HCA)**: 128:1 compression ratio for global context
- **Indexer**: Learned top-k selection of compressed KV positions for sparse attention
- **Attention sink**: Learnable per-head bias to capture global information

### 2. DeepSeekMoE with Hash Routing
- **384 routed experts**, 1 shared expert, top-6 routing
- **Hash routing** for first 3 layers (deterministic expert assignment by token ID)
- **Score-based routing** with `sqrtsoftplus` activation (√softplus) for remaining layers  
- **Auxiliary-loss-free load balancing** via bias correction (`noaux_tc`)
- **SwiGLU with clamping**: `swiglu_limit=10.0` for numerical stability
- **FP4 expert weights** with E8M0 per-32 block scales

### 3. Manifold-Constrained Hyper-Connections (mHC)
- Replaces standard residual connections
- Maintains `hc_mult=4` copies of hidden state
- **Pre-connection**: Sinkhorn-normalized mixing (20 iterations) reduces 4 copies → 1
- **Post-connection**: Expands 1 → 4 copies via learned post-weights + combination matrix
- Improves signal propagation stability across 61 layers

### 4. Multi-Token Prediction (MTP)
- `n_mtp_layers=1`: One additional prediction head
- MTP block has its own transformer block + embedding projection
- Shares embedding and head weights with main model

## 📊 Model Configurations

| Config | Total Params | Active Params | Layers | Experts | Hidden | Heads |
|--------|-------------|---------------|--------|---------|--------|-------|
| V4-Pro | 1.6T | 49B | 61 | 384 | 7168 | 128 |
| V4-Flash | 284B | 13B | 61 | 128* | 4096* | 64* |
| V4-Nano (ours) | ~523M | ~200M | 7 | 8 | 1024 | 16 |

*V4-Flash specs are estimated from the paper.

## 🔧 Key Differences from DeepSeek-V3

| Feature | DeepSeek-V3 | DeepSeek-V4 |
|---------|-------------|-------------|
| Attention | MLA with full KV | MLA + Compressed Sparse Attention |
| KV format | `kv_lora_rank=512` separate nope/rope/v | Unified `head_dim=512` |
| Output projection | Single `wo` | Grouped low-rank `wo_a` + `wo_b` |
| Residual connections | Standard | Manifold-Constrained Hyper-Connections |
| Expert scoring | `sigmoid` | `sqrtsoftplus` (√softplus) |
| Expert routing | Score-based all layers | Hash routing (first 3) + score-based |
| Expert precision | FP8 | FP4 (with E8M0 scales) |
| Context length | 128K | 1M tokens |
| Sliding window | None | 128 tokens |
| KV compression | None | Learned gated pooling (4:1 and 128:1) |
| Sparse attention | None | Indexer-based top-k selection |
| Optimizer | AdamW | Muon |
| Training tokens | 14.8T | 32T+ |

## 📁 Files

```
open-deepseek-v4/
├── configuration_deepseek_v4.py   # HF-compatible config class
├── modeling_deepseek_v4.py        # Full model implementation (pure PyTorch, no custom kernels)
├── architecture_analysis.py       # Detailed V3→V4 architectural comparison
├── test_model.py                  # Comprehensive test suite (12 tests)
├── train.py                       # Training script with HF Trainer
├── configs/
│   ├── config_pro.json            # 1.6T Pro config (matches official)
│   └── config_nano.json           # ~523M Nano config for testing
└── README.md                      # This file
```

## 🚀 Usage

```python
import json
from configuration_deepseek_v4 import DeepSeekV4Config
from modeling_deepseek_v4 import DeepSeekV4ForCausalLM

# Load nano config for testing
with open("configs/config_nano.json") as f:
    config = DeepSeekV4Config(**json.load(f))

model = DeepSeekV4ForCausalLM(config)

# Forward pass
import torch
input_ids = torch.randint(0, config.vocab_size, (1, 128))
outputs = model(input_ids)
logits = outputs.logits  # [1, 128, vocab_size]

# With labels (training)
labels = torch.randint(0, config.vocab_size, (1, 128))
outputs = model(input_ids, labels=labels)
loss = outputs.loss
loss.backward()
```

## ✅ Test Results (Nano config)

All 12 architecture tests pass:
1. ✅ Configuration loading
2. ✅ RMSNorm
3. ✅ RoPE with YaRN scaling
4. ✅ KV Compressor (4:1 and 128:1)
5. ✅ Hyper-Connections (pre + post + head)
6. ✅ MoE Gate (hash + score routing with sqrtsoftplus)
7. ✅ Expert (SwiGLU with clamping)
8. ✅ Full MoE layer
9. ✅ Attention (MLA + Compressed Sparse)
10. ✅ Transformer Block (with HC)
11. ✅ Full model forward + backward
12. ✅ Autoregressive generation

## 📝 Training Recipe (from paper)

- **Optimizer**: Muon (momentum-based, matrix-valued updates)
- **Pre-training**: 32T+ diverse tokens
- **Sequence length**: Progressive — starts shorter, extends to 1M
- **Precision**: FP8 for linear layers (block-wise 128×128), FP4 for experts
- **Post-training**: Two-stage
  1. Independent domain expert cultivation (SFT + RL with GRPO)
  2. Unified model consolidation via on-policy distillation

## ⚠️ Notes

- This is a **pure PyTorch** implementation — no custom CUDA kernels required
- The official inference uses `tilelang` kernels for FP4/FP8 GEMM and sparse attention
- For production-scale training, you'll need the custom kernels from the official repo
- The Nano config is designed for architecture validation, not for meaningful language modeling

## License

MIT (matching DeepSeek-V4's license)

## Citation

```bibtex
@misc{deepseekai2026deepseekv4,
    title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
    author={DeepSeek-AI},
    year={2026},
}
```