---
license: other
license_name: deepseek
license_link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash
base_model: deepseek-ai/DeepSeek-V4-Flash
tags:
  - gale
  - 1-bit
  - quantization
  - mixture-of-experts
  - deepseek
---

# DeepSeek-V4-Flash — GALE 1-bit experts

A 1-bit re-quantization of DeepSeek-V4-Flash's routed experts, built to fit **entirely inside
consumer VRAM**. On the machine it was made on — 2×RTX 3090 + 2×RTX 2080 Ti, 70 GB of VRAM total —
all 11,008 experts are pinned in VRAM with no CPU tier, no overflow tier and no disk reads during
decode. That residency, not the kernels, is what makes it usable.

**159.6 GB → 55.0 GB**, and the same quantizer that scores 0.52–0.54 expert output error on
V4-Pro scores **0.357** here: Flash's smaller experts take 1-bit roughly twice as well.

## What is in the pack

| Weights | Format on disk | Size | Bits/weight |
|---|---|---|---|
| Routed experts (43 layers × 256 experts × w1/w3 [2048,4096] + w2 [4096,2048]) | GALE 1-bit: sign bit per weight under a blocked FWHT-1024 rotation, fp16 scale per 128-weight block, 1% outliers as int4 with a per-row fp16 scale | **46.0 GB** | **1.329** |
| Attention, shared experts, norms, routers, HC-mixing, embedding, head | fp8 (block-scaled) and fp16 on disk; fp8 entries are quantized to 4-bit at load and kept 4-bit in VRAM | 8.9 GB | 8 on disk / 4 in VRAM |

277.0 B routed-expert parameters across 33,024 expert matrices, plus 1,228 non-expert tensors.

## How the experts were quantized

Same v4 recipe as the [V4-Pro pack](https://huggingface.co/benjzzz/DeepSeek-V4-Pro-671B-GALE-1bit-v4):
each sign is chosen with GPTQ-style **error feedback in output space**, weighted by the inverse
Hessian of real activations, with a least-squares per-row gain folded into the block scales at zero
extra bits.

What differs here is the calibration data. V4-Pro's card names "re-calibrate on code" as its next
lever, so this pack was calibrated on **49,152 tokens (24 × 2048) of ~64 % Python source plus
technical English**, captured layer by layer during a real forward pass with the *original* weights,
with each layer's hidden state chained through the *quantized* reconstruction of the layer before it.

The whole 43-layer run took **7 h 07 min** on one RTX 3090 (~10 min/layer, 256 experts each).

## Accuracy

Held-out expert **output** relative error, measured per expert during the run (33,024 matrices):

| | expert_out_rel |
|---|---|
| **whole model (mean)** | **0.3571** |
| median | 0.3537 |
| best layer (layer 0) | 0.244 |
| worst layer (layer 27) | 0.413 |
| *V4-Pro, same quantizer and setting* | *0.52–0.54* |

Error grows with depth, because each layer is calibrated on activations that already carry the
previous layers' quantization error:

`L0 0.244 · L7 0.343 · L14 0.355 · L21 0.344 · L28 0.389 · L35 0.397 · L42 0.338`

**Do not quote the layer-0 number as the model's error.** A single-layer probe reads 0.244 and is
what the bit-width decision was made on; the finished model is 0.357.

### Why 1 bit and 1 % outliers

Both were chosen from measurements on real Flash experts, not assumed. The outlier budget has sharp
diminishing returns, and everything above 1 % costs more VRAM than the accuracy is worth once the
goal is full residency (layer-0 probe):

| outlier budget | expert_out_rel | bits/weight | experts total | fits 70 GB VRAM |
|---|---|---|---|---|
| **1 %** | **0.248** | **1.33** | **46 GB** | **yes, with room for KV cache** |
| 2 % | 0.236 | 1.52 | 53 GB | yes |
| 3 % | 0.227 | 1.73 | 60 GB | tight |
| 5 % | 0.211 | 2.12 | 74 GB | no |

A 2-bit/4-bit alternative (BenQ quantizer, `n_tiers` 2 and 8) is genuinely more accurate — held-out
out_relerr 0.128 / 0.135 / 0.217 for w1/w3/w2 at 2-bit, 0.049 / 0.052 / 0.090 at 4-bit — but lands
at ~96 GB, which spills ~27 % of experts back onto a CPU tier and costs more speed than the accuracy
buys. It also has no serving kernel here. **1-bit was chosen for residency, not because it is more
accurate.**

## Running it

```
pip install "torch>=2.5" triton transformers tokenizers fastapi uvicorn jsonschema
ulimit -n 65536
./serve.sh                      # edit D= at the top to point at the clone
```

`serve.sh` carries every setting that matters. Three are not optional:

- **`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`** — full residency fills each GPU to ~90 %,
  where the default allocator fragments and OOMs with GBs still free.
- **`DSV4_ONESHOT` / `DSV4_ONESHOT_TOOLS`** — see *Prompting* below. These are load-bearing.
- **`DSV4_TIER_BIAS=`** (empty) — the tiering knobs are meaningless when nothing is on a slow tier.

The server plans placement itself from what it finds: it enumerates every visible GPU, reads each
one's free memory, and splits the layers in contiguous blocks **proportional to each device's
budget**. On this mixed set that yields 16/6/15/6 layers and 20.6 / 7.4 / 18.4 / 7.6 GB. If the
model does not fit resident, it falls back to a tiered plan with a CPU expert tier — the same code
serves both.

Serving this pack surfaced four V4-Pro constants hardcoded in that "generic" server, all fixed in
`server/deepseek_v4pro_server_v2.py`: expert-slot sizing looped `range(61)` (Pro's layer count, an
instant `KeyError` on 43); the cache warmers defaulted to 384 experts (Flash has 256); full-residency
placement filled the largest GPU first and left the last card empty; and `RESERVE` was a literal
0.85 despite being documented as `DSV4_RESERVE`.

## Measured performance

2×RTX 3090 + 2×RTX 2080 Ti, Ryzen 9 3900X, 62 GB RAM, weights on NVMe:

| | V4-Flash 1-bit (this pack) | V4-Pro 1-bit, same box |
|---|---|---|
| plan | **full residency**, 11,008 experts pinned, no CPU tier | tiered, ~60–90 of 366 experts/token on the CPU tier |
| load | **80 s** | 130 s |
| decode | **3.4–4.1 tok/s** | 1.2–1.8 tok/s |
| experts touched per token | 258 (43 × 6), all from VRAM | 366, most off CPU/NVMe |
| agent task: write `hello.py`, run it, report output | **196 s, completed** | 250 s |
| VRAM | 20.6 / 7.4 / 18.4 / 7.6 GB | 23.4 / 21.7 / 8.8 / 8.8 GB |

Prefix caching is on by default: the unchanging preamble of a conversation is prefilled once,
snapshotted to `kv_snapshots/` and reused, and a rolling checkpoint every 8 decode tokens lets a
follow-up turn resume mid-answer.

## Prompting (read this before judging the quality)

This model follows instructions weakly enough that **the one-shot example in the preamble decides
what it does**, and getting it wrong looks like a broken model. Measured, on the same prompts:

| one-shot shown | behaviour |
|---|---|
| greeting only | echoes the user's prompt back instead of answering |
| greeting + a markdown code answer | good code, but it *describes* tool work instead of calling tools |
| a tool call only | calls a tool for everything — tries to `read_file` a file that does not exist when merely asked to show a snippet |
| ends after one tool result | does the first step of a task and stops |
| **greeting + terse answer + code answer** (no tools) and **+ a two-step tool example** (tools) | what ships: answers directly, calls tools when asked, and chains steps |

The shipped `DSV4_ONESHOT_TOOLS` demonstrates a *two-step* task whose second step inspects a file.
An otherwise identical example whose second step *runs* the file made the model fabricate the run's
output instead of calling anything — so the bridge also strips `<tool_result>` blocks and stray
`Assistant:` markers from model output, terminated or not.

## Files

- `packed_v4/` — expert pack, `layer_LLL.h0.pack` (43 files) + `manifest_mixed_full.jsonl`
  (34,252 records: every expert matrix and every non-expert tensor, with kind, shape, sections,
  outlier count, pack file and byte offset).
- `misc_v3/` — non-expert tensors, `misc_model-000NN-of-00046.pack` (46 files).
- `server/` — everything needed to run standalone: the engine (`deepseek_v4pro_server_v2.py`) and its
  kernels (`dsv4_kernels.py`, `dsv4_w4.py`, `dsv4_cpu_expert.c` + compiled `.so`, `dsv4_cpu_tier.py`,
  `dsv4_device_plan.py`), the reference implementation (`reference/`, from
  `deepseek-ai/DeepSeek-V4-Flash`; byte-identical to V4-Pro's, the reference code is genuinely
  config-driven), the tokenizer, the quantizer used to build this pack (`v4pro_gptq1bit.py`,
  `v4pro_stream_quantize_v3.py`), and the tool-calling bridge under `server/agent_bridge/`.
- `serve.sh` — the launcher, including the one-shot examples above. It resolves its own directory, so
  a fresh clone runs with no edits: `./serve.sh`.
- `routing_counts.json` — uniform placeholder. Full residency pins everything, so hot-set ranking
  is unused here; it matters only if you run this pack tiered on a smaller machine.

## Honest limits

- **It is not a strong coding model.** It writes a correct iterative Fibonacci where V4-Pro produced
  prose, answers arithmetic and shell-command questions correctly, and drives a real write-then-run
  tool loop end to end. It also still gets things wrong that a good model would not — asked for a
  one-liner counting lines in `.py` files it returns `ls *.py | wc -l`, which counts files.
- **It fabricates tool results** if allowed to keep generating past a tool call. The bridge ends the
  turn at the first closed tool block and strips invented results; do not remove that.
- **Context is 4096 tokens**, and a tool-schema preamble costs ~1,100 of them. Prefix caching makes
  that cheap in time, not in context.
- Each agent step is a full round trip, so multi-step tasks run in minutes, not seconds.
- Quality was measured on a handful of prompts and per-expert reconstruction error, **not** on a
  standard benchmark suite. Treat every quality claim here as indicative.
- Kernels are Triton (any CUDA GPU); the optional fast paths verify themselves at startup and fall
  back to a portable implementation if they cannot run. Nothing runs on Apple Silicon or ARM as-is.
- The expert files were built from the official `deepseek-ai/DeepSeek-V4-Flash` weights streamed
  shard by shard; the original weights are not included here.