Kimi-K2.5-0.5B

This is a tiny version of moonshotai/Kimi-K2.5 created for testing and development.

Model Details

• Base Model: moonshotai/Kimi-K2.5
• Architecture: kimi_k25 (KimiK25ForConditionalGeneration — VLM with DeepSeek V3 MoE text model + vision tower)
• Total Parameters: 0.464B
• Activated Parameters: ~0.04B (text-only, excluding embeddings; MoE with 4/8 experts per token)

Configuration Changes

The following parameters were reduced from the original model:

Parameter	Original	Tiny
text_config.num_hidden_layers	61	3 (1 dense + 2 MoE)
text_config.hidden_size	7168	1024
text_config.intermediate_size	18432	2048
text_config.moe_intermediate_size	2048	512
text_config.num_attention_heads	64	8
text_config.num_key_value_heads	64	8
text_config.q_lora_rank	1536	256
text_config.kv_lora_rank	512	128
text_config.qk_nope_head_dim	128	64
text_config.qk_rope_head_dim	64	32
text_config.qk_head_dim	192	96
text_config.v_head_dim	128	64
text_config.n_routed_experts	384	8
text_config.num_experts_per_tok	8	4
vision_config.num_hidden_layers	27	2
vision_config.vt_num_hidden_layers	27	2
text_config.quantization_config	int4 pack-quantized	removed (unquantized)

Architecture-preserving choices: first_k_dense_replace=1 (1 dense layer then MoE), MLA attention (multi-head latent attention with LoRA-style KV compression), sigmoid scoring + noaux_tc routing, shared experts, vision tower + patchmerger projector.

Checkpoint Structure

Single-file checkpoint (model.safetensors) with 126 tensors. Weight naming matches the original model structure:

• language_model.model.* — text transformer (DeepSeek V3 with MLA + MoE)
• vision_tower.* — vision encoder (MoonViT3d)
• mm_projector.* — vision-to-text projection (patchmerger)

The only structural difference from the original: expert weights are stored as .weight (unquantized float) instead of .weight_packed / .weight_scale / .weight_shape (4-bit compressed).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/Kimi-K2.5-0.5B",
    trust_remote_code=True,
    attn_implementation="eager",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "inference-optimization/Kimi-K2.5-0.5B",
    trust_remote_code=True,
)

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20, use_cache=False)
print(tokenizer.decode(output[0]))

Note: attn_implementation="eager" is required for compatibility with transformers v5. use_cache=False is recommended until KV cache compatibility with transformers v5 is resolved.

Creation Process

This model was created using the llm-compressor create-tiny-model Claude skill.

Steps:

1. Inspected the Kimi-K2.5 config via AutoConfig.from_pretrained
2. Reduced architecture dimensions (layers, hidden size, experts, MLA ranks)
3. Applied transformers v5 compatibility patches (removed DynamicCache.from_legacy_cache, is_torch_fx_available, fixed MoE training-mode forward, tie_weights signature)
4. Fine-tuned on copypasta toy dataset until perplexity < 3.0
5. Validated checkpoint structure and model inference

Validation Output

Success: 1.0005239248275757 <= 10.0
According to all known laws of aviation, there is no way a bee should be able to fly.

Notes

• This is a VLM (vision-language model). Text-only inference works by passing input_ids only (no pixel_values). The vision tower is present but not invoked for text-only tasks.
• The model requires trust_remote_code=True due to custom modeling code for KimiK25 and DeepSeek V3 architectures.
• The modeling files include patches for transformers v5 compatibility (the base moonshotai/Kimi-K2.5 model code targets transformers v4).
• Expert weights are unquantized (bf16) in contrast to the original INT4 pack-quantized weights.