Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid

Qwen3.8-Flash-Next W4A16 AutoRound hybrid (int4 + int8 lm_head + fp8 side layers)

Ready-to-serve checkpoint for Saren-Arterius/qwen3.8-Flash-DGX-AutoRound — Qwen3.8-Flash-Next on a single DGX Spark (GB10) at ~49 tok/s with MTP, prefix caching and never-evict prompt pinning.

This is Intel/Qwen3.8-Flash-Next-W4A16-AutoRound with the repo's CPU-only tools/ applied:

• bf16 lm_head → int8 GPTQ-Marlin (shared with the MTP draft head)
• 300 bf16 side-layer tensors (GDN/QSA projections, shared expert) → blockwise fp8 e4m3 128×128
• the 51B n-gram ("PLE") table stripped from the index — it is served separately via mmap from Saren/Qwen3.8-Flash-Next-ple-table-fp8
• quantization_config rewritten to GPTQ + dynamic rules (the original AutoRound config is kept as config.json.autoround)

Use

git clone https://github.com/Saren-Arterius/qwen3.8-Flash-DGX-AutoRound
hf download Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid --local-dir /models/Qwen3.8-Flash-Next-W4A16-AutoRound
hf download Saren/Qwen3.8-Flash-Next-ple-table-fp8 --local-dir /models/ple-table-fp8
# then: docker build + serve.sh — see the GitHub README

Weights carry Qwen's license (MAU/revenue clause) — review before production use.

Variant: int4 MTP draft experts

Saren/Qwen3.8-Flash-Next-W4A16-AutoRound-hybrid-MTP_int4RTN is this checkpoint with the MTP drafter's 512 routed experts (bf16 here, ~4.7 GiB) quantized to int4 GPTQ g128 RTN: −3.5 GiB resident, +2–4% decode, same outputs (the target verifies every drafted token), draft acceptance unchanged. Same PLE table.