jev-decisions-v2-model-gguf

GGUF exports of GhostScientist/jev-decisions-v2-model (Jev/Tev1-style single-letter decision model: full fine-tune of Qwen3.5-0.8B on GhostScientist/jev-decisions-v1).

file	quant	size
jev-decisions-v2-q4_k_m.gguf	Q4_K_M	529 MB
jev-decisions-v2-q8_0.gguf	Q8_0	812 MB

Converted with llama.cpp convert_hf_to_gguf.py --no-mtp (required for qwen3_5_text: the config declares an MTP layer with no tensors, and the default conversion writes a 25-block metadata header that llama-server rejects with check_tensor_dims: tensor 'blk.24.attn_norm.weight' not found).

Measured on this exact Q4_K_M export path:

eval	accuracy	valid-letter rate	latency
llama.cpp, 4 CPU threads, 200 dev samples, greedy	0.82	1.00	p50 ~1.33 s, p95 ~2.47 s
Ollama (/api/chat), 40 dev samples	0.925	1.00	p50 ~2.6 s
GPU bf16 reference (400 dev samples)	0.80	1.00	mean ~138 ms

(Ollama ran on the first 40 dev rows only — not directly comparable to the full-split numbers.)

Ollama

huggingface-cli download GhostScientist/jev-decisions-v2-model-gguf \
    jev-decisions-v2-q4_k_m.gguf --local-dir .
ollama create jev-decisions-v2 -f Modelfile
ollama run jev-decisions-v2 "..."

The bundled Modelfile points FROM at the local Q4_K_M file, bakes in the decision system prompt, and pins temperature 0 / num_ctx 4096. Send the task as a user message containing {"state": ..., "question": ..., "options": [...]} JSON.

Alternatively, a Modelfile with FROM hf.co/GhostScientist/jev-decisions-v2-model-gguf:Q4_K_M pulls the GGUF straight from the Hub (Ollama >= 0.7; note FROM accepts a model reference or a local path — not a raw https URL).

Serving notes (read this before integrating)

The Qwen3.5 chat template defaults to thinking mode, and this model answers with a bare letter (it never emits a think-marker). Where the response ends up depends on the server's template handling:

• llama-server: the letter can be routed to reasoning_content with an empty content. Prefer chat_template_kwargs: {"enable_thinking": false} — measured there, accuracy went 0.00 → 0.82 on the same weights.
• Ollama (verified): the letter lands in message.thinking with an empty content. Pass "think": false in the /api/chat payload (Ollama >= 0.9), or fall back to reading message.thinking when content is empty.

A robust client therefore reads content, then thinking/reasoning_content, then applies the letter regex to whichever is non-empty.