bayliner1980/Qwen2.5-0.5B-MoE-A0.3B

Qwen2.5-0.5B-MoE-A0.3

A sparse mixture-of-experts model derived from Qwen/Qwen2.5-0.5B. Each token uses about 40% of the original MLP compute: 8 of 32 routed experts plus a shared expert in each MoE layer. Attention, embeddings and the tokenizer are the same as Qwen2.5-0.5B.

It is a stock Qwen2MoeForCausalLM, so it loads with plain transformers and needs no custom code:

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("bayliner1980/Qwen2.5-0.5B-MoE-A0.3B")
model = AutoModelForCausalLM.from_pretrained("bayliner1980/Qwen2.5-0.5B-MoE-A0.3B", dtype="bfloat16")

inputs = tok("The history of the printing press", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=50)
print(tok.decode(out[0], skip_special_tokens=True))

Architecture

Layers	24: layers 0-22 are MoE, layer 23 is a dense MLP (mlp_only_layers=[23])
Routed experts	32 per MoE layer, top-8, norm_topk_prob=True
Expert size	128
Shared expert	768, sigmoid-gated
Active MLP compute	37% of the dense model's in each MoE layer, 39.5% averaged over all layers
Active parameters	~168M of 358M non-embedding parameters per token
Total parameters	494.7M (bf16, ~990 MB)
Context length, vocabulary	Same as Qwen2.5-0.5B

Evaluation

Perplexity uses 1024-token windows. Benchmarks are zero-shot and follow lm-evaluation-harness conventions (HellaSwag on 3,000 validation items; full sets for the others).

	WikiText-2 test ppl	C4 val ppl	HellaSwag (norm)	ARC-Easy (acc)	ARC-Challenge (norm)	WinoGrande	OpenBookQA (norm)	Mean acc_norm
Qwen2.5-0.5B (dense)	14.66	21.29	0.497	0.646	0.321	0.569	0.354	0.467
This model	19.20	28.97	0.440	0.555	0.265	0.519	0.308	0.408

Mean acc_norm averages acc_norm over all five tasks (ARC-Easy acc_norm: 0.508).

Changes from the previous version

The previous upload scored better on WikiText-2 (17.99) but worse on general web text (C4 ~35) and benchmarks (mean acc_norm 0.393). This version is more general-purpose. It uses slightly more compute per token than before (39.5% vs 37% of the MLP) because its last layer is dense.

Speed

The sparsity only saves compute with an inference engine that runs just the selected experts, such as transformers' MoE kernels, vLLM or llama.cpp. At this model size attention and the embedding/LM head make up a large share of the cost, so expect smaller end-to-end speedups than the MLP figures suggest.

Limitations

• Behind the dense model on reasoning and knowledge. It keeps most of the dense model's language-modelling quality, but loses about 0.06 mean benchmark accuracy. Expect factual errors and topic drift in long generations.
• Code and non-English text not evaluated. Quality there is likely weaker than on English prose.
• Base model, not instruction-tuned. It continues text; it does not follow chat instructions.
• Research artefact. It is an experiment in sparse conversion, not a model for production use.

Support

This model was built entirely on my own hardware, as an independent experiment. The results are promising, and there's a lot more I'd like to try. Converting larger models, training for longer and testing more configurations should all improve quality, but they need far more compute than my current hardware can provide. With your support, I can rent cloud GPUs to run these experiments and release better and larger models. Any new models will be shared openly here. If you find this work useful or interesting, any contribution helps.

License

Apache 2.0, the same as the base model Qwen/Qwen2.5-0.5B.