Qwen2.5-1.5B-MoE-A0.9B

A Mixture-of-Experts version of Qwen2.5-1.5B, converted from the dense model without pre-training from scratch. Every MLP layer was split into an always-on shared expert plus 32 routed experts, and each token uses 8 of them. That leaves ~0.85B active parameters per token instead of 1.54B. Every layer is MoE, so the model runs in stock llama.cpp with no patches. In llama.cpp it generates text 32% faster than the dense model on GPU and 41% faster on CPU.

GGUF files (llama.cpp, LM Studio, Ollama): Qwen2.5-1.5B-MoE-A0.9B-GGUF

This is a base (completion) model, like the original. It is not instruction-tuned.

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

path = "bayliner1980/Qwen2.5-1.5B-MoE-A0.9B"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype="bfloat16", device_map="auto")

ids = tok("The history of the printing press", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=80, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

It loads as a standard Qwen2MoeForCausalLM; no custom code or trust_remote_code is needed.

Architecture

	Qwen2.5-1.5B (dense)	This model
Layers / hidden size / heads	28 / 1536 / 12 (2 KV)	same
MLP per layer	8960 neurons, all active	shared expert 1792 + 32 experts × 224, top-8
Active MLP neurons per token	8960	3584 (40%)
Active parameters per token*	1.54B	0.85B
Total parameters	1.54B	1.78B
Output head	tied to embeddings	untied (llama.cpp's qwen2moe requires a separate output tensor)

* Active parameters exclude the embedding lookup and include the output head. The total grows only because the output head is untied; the experts together hold the same number of neurons as the original MLP.

The routing uses norm_topk_prob: false, the only mode llama.cpp's qwen2moe supports. The expert width (224) is a multiple of 32, so every expert tensor quantizes with llama.cpp's k-quants.

Quality

Perplexity: lower is better. Tasks: zero-shot acc_norm, higher is better, on the first 1000 items of each benchmark (500 for OpenBookQA), with the dense model scored the same way.

	Dense Qwen2.5-1.5B	This model
WikiText-2 test perplexity (1024-token windows)	10.41	14.63 (+41%)
C4 validation perplexity (64 × 1024 tokens)	15.76	22.88 (+45%)
Zero-shot tasks, mean acc_norm	0.565	0.462

Task	Dense	This model
HellaSwag	0.592	0.510
ARC-Easy	0.723	0.572
ARC-Challenge	0.452	0.311
WinoGrande	0.646	0.588
OpenBookQA	0.410	0.328

Speed

llama.cpp llama-bench, Q4_K_M for both models, tokens per second (RTX 4060 8 GB; Ryzen 5 7500F for CPU):

	Dense Qwen2.5-1.5B	This model	Change
GPU, text generation	191	253	+32%
GPU, prompt processing (512 tokens)	10,329	11,353	+10%
CPU, text generation	54.9	77.5	+41%
CPU, prompt processing (512 tokens)	3,615	3,646	≈ same

The GGUF files are larger than the dense model's (Q4_K_M: 1125 vs 935 MiB) because of the untied output head.

How it was made

The dense model's MLP neurons were grouped into a shared expert and routed experts, a router was fitted for each layer. Everything ran on a single 8 GB consumer GPU. No layer was kept dense, so the model fits llama.cpp's qwen2moe architecture as is.

Limitations

• About 41–45% higher perplexity than the dense original, and clearly weaker on reasoning benchmarks (ARC in particular). Use it where speed matters more than the last bit of quality.
• A base model: it continues text and does not follow instructions or chat reliably.
• Distilled on English educational web text only. Other languages and code may have degraded more than the English numbers suggest.
• Inherits the biases and limitations of Qwen2.5-1.5B.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen2.5-1.5B.