peonist-ai

/

halogen-qwen3.8-flash-next

Like 170

Follow

Peonist 126

This is a great idea and yes it is possible. I am working on opening up the harness to allow anyone to onboard new models onto the engine and auto tune the kernels. Stay tuned!

Awesome! Looking forward to it, feel free to ping here when you need another tester (I’m running a Beelink GTR9 in case it matters)

This is a great idea and yes it is possible. I am working on opening up the harness to allow anyone to onboard new models onto the engine and auto tune the kernels. Stay tuned!

This sounds great!
Thank you for your awesome effort to make the Strix Halo the horsepower it deserves to be.

I would probably like to contribute then by converting (and of course also uploading) orcarouter/qwen3.8-flash-next as I used it heavily and liked it the most.

You should be able to BYOGGUF with the latest update:

https://github.com/peonist-ai/halogen-flash-server#bring-your-own-gguf

Hello!
Is there any "uncensored" gguf model suggestion? I'm not sure which model can be used with BYO GGUF constrain. I really appreciate your help!

I have never tried one, but this seems popular. Maybe check out https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF

IQ4_XS should work. Check the readme in github.

Thanks anyway! I will try this later!

sadly the IQ4_XS of orcarouter doesnt work
engine-1 | halogen: /models/Qwen3.8-Flash-Next-Uncensored/IQ4_XS/Qwen3.8-Flash-Next-Uncensored-IQ4_XS-00001-of-00003.gguf is a GGUF: it is repacked into RAM at startup, losslessly, on every start
engine-1 | (about 20 s from a cold NVMe disk on the reference machine, 9 s warm; HALOGEN_GGUF_CACHE=1 keeps
engine-1 | a copy beside it and a warm restart is then about 1 s) and served with the engine's own draft
engine-1 | head. The quality sidecar does not apply to a GGUF trunk.
engine-1 | halogen: draft head /models/Qwen3.8-Flash-Next-Uncensored/IQ4_XS/qwen38-flash-next-mtp.hgn (1.5G)
engine-1 | halogen: vision ON, tower /models/Qwen3.8-Flash-Next-Uncensored/IQ4_XS/qwen38-flash-next-vision.hgn (857M)
engine-1 | an image costs 1,000 tokens at 1280x800 and ~2,040 at 1080p;
engine-1 | HALOGEN_VISION_MAX_PIXELS caps it (default 2560x1440; larger is downscaled).
engine-1 | halogen: KV budget 4 slot(s) over one 524288-position pool (each request up to 262144) = 14.9 GiB (28 KiB/position incl. block scratch + ~115 MiB/slot) + 1.8 GiB prompt cache in RAM, on top of roughly 72 GiB (an 8-bit GGUF trunk, repacked into RAM) of weights and 11 GiB of scratch. Those last two are estimates for this pre-flight check; the engine prints its measured figures once loaded, including the large lookup table it reads from disk and never holds. MemAvailable now 122.5 GiB.
engine-1 | gguf: blk.1.ple_conv1d.weight: F16 is not read by this engine (Q4_1, Q5_0, Q2_K, Q3_K and the IQ2/IQ1/F16 families are not repacked losslessly; no lossy fallback is offered). Use a build whose experts are IQ4_NL, IQ4_XS, IQ3_S, Q4_0, Q4_K, Q5_K or Q5_1 and whose trunk is Q8_0, such as unsloth's UD-IQ4_XS or UD-Q4_K_XL
engine-1 | halogen: the engine exited after 2s without listening.
nevermind, works with 0.12.1

Just put out a 0.12.1 update. Try again. 😀

Would be so cool to have peonist-ai/halogen-qwen3.8-flash-next-uncensored 🥸

Just put out a 0.12.1 update. Try again. 😀

yes works perfectly fine. Now the only thing left is missing rpc support :), got 2 of them and higher quants would be great, but impressive benchs thx

Was just stopping in to say that I have been able to get my own IQ4XS quant of orcarouter/Qwen3.8-Flash-Next-Uncensored running on Halogen with 1k+ tps prefill and 35+ tps decode. Using Open WebUI that is a time to first token of about 6seconds down from 30+ seconds on llamacpp!

Was just stopping in to say that I have been able to get my own IQ4XS quant of orcarouter/Qwen3.8-Flash-Next-Uncensored running on Halogen with 1k+ tps prefill and 35+ tps decode. Using Open WebUI that is a time to first token of about 6seconds down from 30+ seconds on llamacpp!

how many slots, which variables did you increase, dont get that guide and even defaults are unclear

I am uploading a copy of the gguf and the hgn as we speak. In the meantime....

• I used the safe-tensor copy of orcarouter/Qwen3.8-Flash-Next-Uncensored
• converted it to a bf16 gguf
• used the imatrix from unsloth/Qwen3.8-Flash-Next-GGUF
• llama-quantize to produce an IQ4_XS variant

I am not sure what you are asking otherwise. I am not an expert so this is just the results from me experimenting and asking questions. I think the trick was quantizing my own version of the Orca model using Unsloth imatrix. I found that the Orca models on offer would not load.

Working uncensored model, based on the orarouter safe-tensors: https://huggingface.co/davenetdev/halogen-qwen3.8-flash-next-uncensored

Amazing! Thank you for this. First custom HGN model in the wild!

Working uncensored model, based on the orarouter safe-tensors: https://huggingface.co/davenetdev/halogen-qwen3.8-flash-next-uncensored

Working uncensored model, based on the orarouter safe-tensors: https://huggingface.co/davenetdev/halogen-qwen3.8-flash-next-uncensored

you could convert it as hgn to speed up the launch, for example in my case
docker run --rm
--device /dev/kfd --device /dev/dri --group-add video
--ipc=host --ulimit memlock=-1:-1
-e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
-v /opt/halogen/models:/models
ghcr.io/peonist-ai/halogen-flash-server:0.12.1
convert /models/Qwen3.8-Flash-Next-Uncensored/IQ4_XS/Qwen3.8-Flash-Next-Uncensored-IQ4_XS-00001-of-00003.gguf /models/flash-next-iq4xs-uncensored.hgn

just a question, why didnt you use orcarouter's gguf, which was working out of the box. I get around 40t/s with 1M context

I thought thats what @davenetdev posted... then I opened the files on his model card and realized it was GGUF! Yes, sorry I'm not sure if the documentation was clear (its getting very log in github). You can output HGN models for anyone that isn't aware.

@Ceynt when I tried orcarouter's gguf it would not load for me. That said, one of the recent Halogen updates may have fixed that. Also, the hgn file is uploading now.

When I try to load the converted hgn i get...

checkpoint: no tensor named layers.1.ple.ngram_embedding.weight

When I try to load the converted hgn i get...

checkpoint: no tensor named layers.1.ple.ngram_embedding.weight

do you use the recent version? i could load the orcarouter gguf without any changes also the hgn of said model. Running on 2 servers atm

I am going to give it another try. Sounds like you are getting better decode speeds than I am so it is work downloading and trying again.

When I try to load the converted hgn i get...

checkpoint: no tensor named layers.1.ple.ngram_embedding.weight

Investigating... think I found the issue. Will be in the next release, hopefully tonight if I don' t fall asleep at my keyboard.

To be clear...

checkpoint: no tensor named layers.1.ple.ngram_embedding.weight

Happens only when I try to load the hgn file converted from my working gguf. This is not the issue I had when trying to load the orcarouter model. I suspect that orcarouter issue is a thing of the past now.

You are doing amazing work @peonist !!!

AI generated, human reviewed.

Thanks, and congratulations on the first custom checkpoint in the wild.

I read the header of the .hgn you uploaded: it carries 1166 tensors, no n-gram lookup table and no draft head. A complete converted checkpoint has 1198. That is the shape flash_serve --repack IN.gguf --out OUT.hgn produced without --with-table (and with --no-head) on 0.12.2 and earlier. Your GGUF has the table, since it serves; the repack left it out because the 0.7.0 changelog and the command's usage printed --with-table in brackets as if it were optional, and it was not. That is our documentation bug.

Fixed in 0.12.3, out now: --repack writes the table by default, --no-table is the opt-out, and the loader's message for that tensor says what happened. The shorter way on any version is the image's convert, which always folded in the table and the draft head:

docker run --rm --device /dev/kfd --device /dev/dri --group-add video --ipc=host \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v /path/to/models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.12.3 \
  convert /models/qwen3.8-flash-next-uncensored-IQ4_XS.gguf /models/qwen3.8-flash-next-uncensored-IQ4_XS.hgn

The result is about 106 GB and starts on its own with HALOGEN_CHECKPOINT=/models/qwen3.8-flash-next-uncensored-IQ4_XS.hgn, no GGUF or head file beside it. If you re-upload, that is the file to publish.

One note for anyone quantizing their own: the engine reads the lookup table in IQ4_NL only, which is what llama-quantize produces for it from an IQ4_XS recipe (yours, unsloth's, bartowski's). A Q4_K_M, Q5_K or Q6_K recipe lands that table on Q5_0, Q5_1 or Q8_0 and the engine refuses the file with a message naming the type. Reading those is the next GGUF item.

Nice! That worked. The converted hgn loads and runs as expected. Uploading the new version now.

Nice! That worked. The converted hgn loads and runs as expected. Uploading the new version now.

Had the same idea about uploading the checkpoint. Upload took a long time because of limited upload capabilities, but there you go:
https://huggingface.co/G1LL1/Qwen3.8-Flash-Next-Uncensored-orca-halogen

Nice! That worked. The converted hgn loads and runs as expected. Uploading the new version now.

Had the same idea about uploading the checkpoint. Upload took a long time because of limited upload capabilities, but there you go:
https://huggingface.co/G1LL1/Qwen3.8-Flash-Next-Uncensored-orca-halogen

Amazing!

The only thing I'd ask is to use the latest tag, or let users know to check the github repo version. We are already on 0.12.3 with another bun in the oven.

AI generated, human reviewed.

Thanks, and congratulations on the first custom checkpoint in the wild.

I read the header of the .hgn you uploaded: it carries 1166 tensors, no n-gram lookup table and no draft head. A complete converted checkpoint has 1198. That is the shape flash_serve --repack IN.gguf --out OUT.hgn produced without --with-table (and with --no-head) on 0.12.2 and earlier. Your GGUF has the table, since it serves; the repack left it out because the 0.7.0 changelog and the command's usage printed --with-table in brackets as if it were optional, and it was not. That is our documentation bug.

Fixed in 0.12.3, out now: --repack writes the table by default, --no-table is the opt-out, and the loader's message for that tensor says what happened. The shorter way on any version is the image's convert, which always folded in the table and the draft head:

docker run --rm --device /dev/kfd --device /dev/dri --group-add video --ipc=host \
  -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next \
  -v /path/to/models:/models \
  ghcr.io/peonist-ai/halogen-flash-server:0.12.3 \
  convert /models/qwen3.8-flash-next-uncensored-IQ4_XS.gguf /models/qwen3.8-flash-next-uncensored-IQ4_XS.hgn

The result is about 106 GB and starts on its own with HALOGEN_CHECKPOINT=/models/qwen3.8-flash-next-uncensored-IQ4_XS.hgn, no GGUF or head file beside it. If you re-upload, that is the file to publish.

One note for anyone quantizing their own: the engine reads the lookup table in IQ4_NL only, which is what llama-quantize produces for it from an IQ4_XS recipe (yours, unsloth's, bartowski's). A Q4_K_M, Q5_K or Q6_K recipe lands that table on Q5_0, Q5_1 or Q8_0 and the engine refuses the file with a message naming the type. Reading those is the next GGUF item.

Works perfectly! halogen-flash-server:0.12.3 on Strix Halo 128Gb.

Nice! That worked. The converted hgn loads and runs as expected. Uploading the new version now.

Had the same idea about uploading the checkpoint. Upload took a long time because of limited upload capabilities, but there you go:
https://huggingface.co/G1LL1/Qwen3.8-Flash-Next-Uncensored-orca-halogen

Amazing!

The only thing I'd ask is to use the latest tag, or let users know to check the github repo version. We are already on 0.12.3 with another bun in the oven.

Sure, no problem. Updated the README to 0.12.3

Nice! That worked. The converted hgn loads and runs as expected. Uploading the new version now.

Had the same idea about uploading the checkpoint. Upload took a long time because of limited upload capabilities, but there you go:
https://huggingface.co/G1LL1/Qwen3.8-Flash-Next-Uncensored-orca-halogen

Amazing!

The only thing I'd ask is to use the latest tag, or let users know to check the github repo version. We are already on 0.12.3 with another bun in the oven.

Sure, no problem. Updated the README to 0.12.3

...and, now we are on 0.13.0. You could just use the latest tag 😀

Fu**, man! You're fast! 🤤

Fu**, man! You're fast! 🤤

Go check out the new tools... I think you will like them 😃

Man, I can't keep the pace. So I just automated the process...

#!/usr/bin/env bash
# halogen.sh — run latest halogen-flash-server, prune old images
set -euo pipefail

REGISTRY="ghcr.io"
REPO="peonist-ai/halogen-flash-server"
IMAGE="${REGISTRY}/${REPO}"
MODEL_DIR="/mnt/data/models/halogen-qwen3.8-flash-next"
PORT=8731

latest_tag() {
  local token
  token=$(curl -fsSL "https://${REGISTRY}/token?scope=repository:${REPO}:pull" | jq -r .token)
  curl -fsSL -H "Authorization: Bearer ${token}" \
      "https://${REGISTRY}/v2/${REPO}/tags/list?n=1000" \
    | jq -r '.tags[]' \
    | grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$' \
    | sort -V | tail -n1
}

local_latest_tag() {
  podman images --filter "reference=${IMAGE}" --format '{{.Tag}}' \
    | grep -E '^v?[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -n1
}

clean() {
  local keep="$1"
  podman images --filter "reference=${IMAGE}" --format '{{.Tag}}' \
    | grep -vxF "$keep" \
    | xargs -r -I{} podman rmi "${IMAGE}:{}" || true   # skips images still in use
  podman image prune -f >/dev/null                      # dangling layers
}

run() {
  local tag="$1"
  podman pull -q "${IMAGE}:${tag}" >/dev/null
  exec podman run --rm -p "${PORT}:${PORT}" \
    --device /dev/kfd --device /dev/dri --group-add keep-groups \
    --ipc=host --ulimit memlock=-1:-1 \
    -v "${MODEL_DIR}:/models:ro" \
    "${IMAGE}:${tag}"
}

resolve() {
  if [[ -n "${VERSION:-}" ]]; then echo "$VERSION"; return; fi
  latest_tag 2>/dev/null || { echo "registry unreachable, using local image" >&2; local_latest_tag; }
}

case "${1:-run}" in
  run)    tag=$(resolve); echo "using ${IMAGE}:${tag}" >&2; clean "$tag"; run "$tag" ;;
  clean)  tag=$(resolve); clean "$tag" ;;
  latest) resolve ;;
  *)      echo "usage: $0 [run|clean|latest]   (VERSION=x.y.z to pin)" >&2; exit 1 ;;
esac

Attempted to use IQ4_XS-PLEQ4_0 from https://huggingface.co/AesSedai/Qwen3.8-Flash-Next-GGUF and the conversion succeeded but the output is a mixed bag of emoji, Latin char, and Chinese char. I am assuming this is because the quant is a mix of Q8_0 / IQ3_S / IQ3_S / IQ4_XS.

...
halogen: no /tokenizer mount, using /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/tokenizer from the models volume
serve_api: request defaults none set (built-in)
serve_api: chat template /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/tokenizer/chat_template.jinja sha256 c3cf9e34abf4f9e3, thinking control probe passed
halogen: waiting for engine on 8730 (cold load can take minutes)
kv pool: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf is a GGUF; 71.2 GiB of resident weights once repacked (plus the head)
checkpoint: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf is a GGUF (qwen4exp, file_type 7, 3 shards, 1224 tensors): 1166 repacked into 69.78 GiB (a_log 36, cat_bf16 12, f32_bf16 751, gate_up 48, i64 3, q4cp 48, q8s32 268), the lookup table read in place, the MTP head from /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/qwen38-flash-next-mtp.hgn
checkpoint: repack cache /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0.hgn matches the shards; taking the .hgn path
checkpoint: mapped 74.9 GB, NOT pinned (Pin::None)
checkpoint: registered 1.5 GB in 0.1 s (18.8 GB/s, Mapped|ReadOnly), device ptr == host ptr
checkpoint: GGUF ready in 0.1 s: 1198 tensors (1166 trunk, 1 table, 31 head)
overlay: none (no /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS-PLEQ4_0/Qwen3.8-Flash-Next-IQ4_XS-PLEQ4_0-00001-of-00003.gguf.overlay.hgn, HALOGEN_CK_OVERLAY unset): the bare checkpoint runs
...

Follow up:

Interestingly the IQ4_XS quant fails to load with :latest (or is doing so incredibly slowly).

trevorstarick@fedora43 ~ [1]> podman run --rm -p 8731:8731 \
                                    --device /dev/kfd --device /dev/dri --group-add keep-groups --pull=newer \
                                    --security-opt seccomp=unconfined --ipc=host --ulimit memlock=-1:-1 \
                                    -v /home/trevorstarick/models:/models \
                                    -e HALOGEN_VERBOSE=1 \
                                    -e HALOGEN_COMPOSABLE_CONTEXT=1 -e HALOGEN_KV_SLOTS=1 -e HALOGEN_KV_POOL_POSITIONS=262144 \
                                    -e HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next -e HALOGEN_GGUF_CACHE=1 \
                                    -e HALOGEN_CHECKPOINT=/models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/Qwen3.8-Flash-Next-IQ4_XS-00001-of-00005.gguf ghcr.io/peonist-ai/halogen-flash-server:latest
halogen: halogen-flash-server 0.13.2, mode all
halogen: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/Qwen3.8-Flash-Next-IQ4_XS-00001-of-00005.gguf is a GGUF: it is repacked into RAM at startup, losslessly, on every start
         (about 20 s from a cold NVMe disk on the reference machine, 9 s warm; HALOGEN_GGUF_CACHE=1 keeps
         a copy beside it and a warm restart is then about 1 s; `convert` writes it as a standalone .hgn)
         and served with the engine's own draft head. The quality sidecar does not apply to a GGUF trunk.
halogen: fetching the draft head qwen38-flash-next-mtp.hgn from peonist-ai/halogen-qwen3.8-flash-next (1.4 GiB)
Hint: A new version of huggingface_hub (1.32.0) is available! You are using version 1.28.0.
To update, run: hf update
Hint: The `hf-cli` skill is not installed. Run `hf skills add -g --claude` to teach your AI agents how to use the `hf` CLI.
✓ Downloaded
  path: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/qwen38-flash-next-mtp.hgn
halogen: draft head /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/qwen38-flash-next-mtp.hgn (1.5G)
halogen: HALOGEN_GGUF_CACHE=1: the repack is written once beside the GGUF (about 70 GiB for an 8-bit trunk) and read on later starts
halogen: fetching the tokenizer from peonist-ai/halogen-qwen3.8-flash-next
Fetching 6 files:   0%|          | 0/6 [00:00<?, ?it/s]Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Fetching 6 files: 100%|██████████| 6/6 [00:00<00:00, 12.16it/s]
✓ Downloaded
  path: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS
halogen: KV budget 1 slot(s) over one 262144-position pool (each request up to 262144) = 7.3 GiB (~28 KiB/position incl. block scratch + ~115 MiB/slot) + 2.2 GiB prompt cache in RAM, on top of roughly 72 GiB or more (a GGUF trunk of file type 7, repacked into RAM; measured for types 30 and 15 only) of weights and 21.3 GiB of working memory (HALOGEN_MAX_TOK 32768). Those last two are estimates for this pre-flight check; the engine prints its measured figures once loaded, including the large lookup table it reads from disk and never holds. MemAvailable now 121.5 GiB.
halogen: GTT in use before this start: 0.0 GiB of 124.0 (124.0 free; this start puts about 24.0 GiB there)
halogen: no /tokenizer mount, using /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/tokenizer from the models volume
serve_api: request defaults none set (built-in)
serve_api: chat template /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/tokenizer/chat_template.jinja sha256 c3cf9e34abf4f9e3, thinking control probe passed
halogen: waiting for engine on 8730 (cold load can take minutes)
kv pool: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/Qwen3.8-Flash-Next-IQ4_XS-00001-of-00005.gguf is a GGUF; 70.2 GiB of resident weights once repacked (plus the head)
checkpoint: /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/Qwen3.8-Flash-Next-IQ4_XS-00001-of-00005.gguf is a GGUF (qwen4exp, file_type 7, 5 shards, 1224 tensors): 1166 repacked into 68.74 GiB (a_log 36, cat_bf16 12, f32_bf16 389, gate_up 48, i64 3, q4cp 48, q8s32 630), the lookup table read in place, the MTP head from /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/qwen38-flash-next-mtp.hgn
checkpoint: repacking on 8 threads (HALOGEN_GGUF_THREADS)
checkpoint: repacking GGUF, 0 of 1166 tensors, 0.0 of 68.7 GiB, 5 s
checkpoint: repacking GGUF, 8 of 1166 tensors, 7.0 of 68.7 GiB, 10 s
checkpoint: repacking GGUF, 8 of 1166 tensors, 7.0 of 68.7 GiB, 15 s
checkpoint: repacking GGUF, 16 of 1166 tensors, 14.1 of 68.7 GiB, 20 s
checkpoint: repacking GGUF, 16 of 1166 tensors, 14.1 of 68.7 GiB, 25 s
checkpoint: repacking GGUF, 24 of 1166 tensors, 21.1 of 68.7 GiB, 30 s
checkpoint: repacking GGUF, 24 of 1166 tensors, 21.1 of 68.7 GiB, 35 s
checkpoint: repacking GGUF, 32 of 1166 tensors, 28.1 of 68.7 GiB, 40 s
checkpoint: repacking GGUF, 33 of 1166 tensors, 29.0 of 68.7 GiB, 45 s
checkpoint: repacking GGUF, 40 of 1166 tensors, 35.2 of 68.7 GiB, 50 s
checkpoint: repacking GGUF, 44 of 1166 tensors, 38.7 of 68.7 GiB, 55 s
halogen: still loading, 60s elapsed (the engine is alive, state D; GTT in use 0.3 GiB; the startup lines above name the step)
checkpoint: repacking GGUF, 48 of 1166 tensors, 42.2 of 68.7 GiB, 60 s
checkpoint: repacking GGUF, 56 of 1166 tensors, 46.1 of 68.7 GiB, 65 s
checkpoint: repacking GGUF, 64 of 1166 tensors, 49.7 of 68.7 GiB, 70 s
checkpoint: repacking GGUF, 66 of 1166 tensors, 50.6 of 68.7 GiB, 75 s
checkpoint: repacking GGUF, 73 of 1166 tensors, 53.8 of 68.7 GiB, 80 s
checkpoint: repacking GGUF, 81 of 1166 tensors, 57.4 of 68.7 GiB, 85 s
checkpoint: repacking GGUF, 88 of 1166 tensors, 60.5 of 68.7 GiB, 90 s
checkpoint: repacking GGUF, 112 of 1166 tensors, 64.6 of 68.7 GiB, 95 s
checkpoint: repacking GGUF, 296 of 1166 tensors, 67.9 of 68.7 GiB, 100 s
checkpoint: repacked 68.74 GiB in 102.4 s (0.72 GB/s of output)
checkpoint: writing the repack cache /models/AesSedai/Qwen3.8-Flash-Next-GGUF/IQ4_XS/Qwen3.8-Flash-Next-IQ4_XS.hgn (68.7 GiB, HALOGEN_GGUF_CACHE)
halogen: still loading, 120s elapsed (the engine is alive, state D; GTT in use 0.3 GiB; the startup lines above name the step)
halogen: still loading, 180s elapsed (the engine is alive, state D; GTT in use 0.3 GiB; the startup lines above name the step)
halogen: still loading, 240s elapsed (the engine is alive, state D; GTT in use 0.3 GiB; the startup lines above name the step)
halogen: still loading, 300s elapsed (the engine is alive, state D; GTT in use 0.3 GiB; the startup lines above name the step)

it's starting in 20 seconds on 2 halo strix pcs for me. As your parameters look like either you are using apparmor in deny mode or a special setup, I don't think it's a general issue. Maybe you could try docker instead, or better the hgn instead of GGUF

Network halogen_default Creating
Network halogen_default Creating
Network halogen_default Created
Network halogen_default Created
Container halogen-engine-1 Creating
Container halogen-engine-1 Created
Container halogen-api-1 Creating
Container halogen-api-1 Created
Attaching to api-1, engine-1
Container halogen-engine-1 Starting
engine-1 | halogen: context 786432 is past the native 262144: HALOGEN_MAX_TOK 32768 is capped at 16384 here (a larger prefill arena leaves a 1M KV cache no room to stay resident).
Container halogen-engine-1 Started
Container halogen-engine-1 Waiting
engine-1 | halogen: halogen-flash-server 0.13.3, mode engine
engine-1 | halogen: /models/flash-next-iq4xs-uncensored.hgn is a converted GGUF trunk (model id qwen3.8-flash-next-gguf): the quality sidecar does not apply and none is looked for
engine-1 | halogen: vision ON, tower /models/qwen38-flash-next-vision.hgn (resolved from HALOGEN_VISION_TOWER=1)
engine-1 | an image costs 1,000 tokens at 1280x800 and ~2,040 at 1080p;
engine-1 | HALOGEN_VISION_MAX_PIXELS caps it (default 2560x1440; larger is downscaled).
engine-1 | halogen: KV budget 3 slot(s) over one 786432-position pool (each request up to 786432) = 21.9 GiB (28 KiB/position incl. block scratch + ~115 MiB/slot) + 2.0 GiB prompt cache in RAM, on top of roughly 68 GiB of weights and 12.9 GiB of working memory (HALOGEN_MAX_TOK 16384). Those last two are estimates for this pre-flight check; the engine prints its measured figures once loaded, including the large lookup table it reads from disk and never holds. MemAvailable now 120.0 GiB.
engine-1 | halogen: WARNING: that budget is close to or over what this host has free (the engine refuses the last pin under 16 GiB of MemAvailable).
engine-1 | If startup ends in "checkpoint: refusing to pin" or "HIP ... out of memory", lower HALOGEN_MAX_TOK to 16384 (the working memory, about 9 GiB back for about 9% of prefill speed)
engine-1 | or HALOGEN_KV_POOL_POSITIONS (the pool, ~29.5 KiB a position); a 1,048,576-position pool fits only with HALOGEN_MAX_TOK=16384.
engine-1 | halogen: GTT in use before this start: 1.5 GiB of 124.0 (122.5 free; this start puts about 30.3 GiB there)
engine-1 | rope: static YaRN, factor 3 over the native 262144 (attention scale 1.109861); every position's RoPE differs from the unscaled model
engine-1 | checkpoint: mapped 102.0 GB, NOT pinned (Pin::None)
engine-1 | overlay: none (no /models/flash-next-iq4xs-uncensored.overlay.hgn, HALOGEN_CK_OVERLAY unset): the bare checkpoint runs
engine-1 | startup [ 0.0 s] host memory: 19676 contiguous 2 MiB blocks of host memory free (39352 MiB)
engine-1 | startup [ 0.0 s] pinning weights (reads the checkpoint from disk when it is not already in the page cache; this is usually the slowest step)
engine-1 | checkpoint: pinned 68.17 GiB in 2 range(s) in 3.1 s (23.4 GB/s), 68.17 GiB of payload, MemAvailable was 119.6 GiB
engine-1 | startup [ 3.1 s] weights pinned: 19606 contiguous 2 MiB blocks free (39212 MiB), 0 compaction stalls during that step (0 failed)
engine-1 | startup [ 3.1 s] preparing weights
engine-1 | startup [ 3.2 s] preparing weights, layer 0 of 48
engine-1 | startup [ 3.3 s] preparing weights, layer 8 of 48
engine-1 | startup [ 3.3 s] preparing weights, layer 16 of 48
engine-1 | startup [ 3.4 s] preparing weights, layer 24 of 48
engine-1 | startup [ 3.4 s] preparing weights, layer 32 of 48
engine-1 | startup [ 3.4 s] preparing weights, layer 40 of 48
engine-1 | affine trunk: 48 tensors staged to bf16 on the device (1.79 GiB); decode reads them at 16 bits a weight
engine-1 | startup [ 3.5 s] weights ready; reserving working memory
engine-1 | checkpoint: mapped 0.9 GB, NOT pinned (Pin::None)
engine-1 | checkpoint: pinned 0.84 GiB in 1 range(s) in 0.0 s (27.8 GB/s), 0.84 GiB of payload, MemAvailable was 99.2 GiB
engine-1 | startup [ 4.3 s] reserving the KV pool, 786432 positions (about 21.6 GiB)
engine-1 | startup [ 14.7 s] KV pool reserved: 244 contiguous 2 MiB blocks free (488 MiB), 127 compaction stalls during that step (49 failed)
engine-1 | startup [ 14.7 s] model ready
engine-1 | startup [ 14.7 s] reserving 2 more serving slot(s)
engine-1 | slots: 1 -> 3 POOLED, 111.0 MiB each of O(1) state; the KV is ONE pool of 786432 positions (19968.0 MiB) carved into per-request regions
engine-1 | startup [ 18.0 s] preparing the prompt cache
engine-1 | flash_serve: prompt cache ON, resume-anywhere (18 entries, snapshot every request, chunk 32768, 111 MiB each at ctx 786432): a warm answer is NOT bitwise the cold one
engine-1 | flash_serve: prompt cache keeps the KV in place (HALOGEN_CACHE_INPLACE=1): the entry is the O(1) state, the KV rows are never copied; a follow-up turn prefills only its new tokens
engine-1 | startup [ 18.0 s] memory: 68.2 GiB of weights locked in RAM, 21.6 GiB of KV pool, 14.2 GiB of working memory, 103.9 GiB in all
engine-1 | startup [ 18.0 s] memory: the model's 26.8 GiB lookup table is NOT held in RAM. It is read from disk through the file cache as each request needs it, which is why this server holds less than the checkpoint is large
engine-1 | startup [ 18.0 s] host memory left for everything else: 373 contiguous 2 MiB blocks (9.3 GiB total, most of it not contiguous)
engine-1 | startup [ 18.0 s] free(1) and MemAvailable will report about 77.5 GiB instead: the kernel counts this server's locked weights as reclaimable file cache, and they cannot be reclaimed
engine-1 | startup [ 18.0 s] opening the socket
engine-1 | halogen: NOTE this server now holds most of the machine, and 373 contiguous 2 MiB blocks remain. It runs best on a host of its own. If other large processes share this machine they will compete for what is left, and when it runs out BOTH they and this server can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart. Lowering HALOGEN_KV_POOL_POSITIONS gives some back. HALOGEN_FLASH_PIN_TRUNK=0 gives a great deal back and costs several times the decode speed, so it is a last resort.
engine-1 | halogen: NOTE only 9.3 GiB of host memory is left after the weights and the KV pool. The lookup table is read from disk through the page cache and is never held, so this is the cache it gets: under about 10 GiB its rows page in from disk on every long prompt, a prefill takes minutes instead of seconds, and the watchdog can read that stall as a wedge. Lower HALOGEN_KV_POOL_POSITIONS (each 262,144 positions is about 7.2 GiB) or HALOGEN_MAX_TOK, or stop other workloads on this host.
engine-1 | flash_serve: WARNING binding 0.0.0.0: the token protocol has no auth; keep this port unpublished
engine-1 | flash_serve: a prompt admitted beside active streams prefills in pieces of 16384 tokens with a decode tick between them (HALOGEN_ADMIT_CHUNK; 0 = the engine's own chunk, which keeps the admitted prompt bitwise its alone self; -1 = the whole prompt at once)
engine-1 | flash_serve: listening on 0.0.0.0:8730, 3 slots over ONE 786432-position KV pool, each request up to 786432 (it reserves prompt + max_tokens), max_tok 16384, BATCHED greedy or sampled; MTP depth-1 selectable (drafter 1, speculates while alone)
engine-1 | halogen: engine listening after 20s
engine-1 | halogen: engine watchdog on, 180s of silence takes the container down

The orcarouter one works fine but the quant by AesSedai does not. I just realized that this thread isn't for the BYOGGUF but specifically the ablated model, my bad.

Since per_layer_token_embd.weight is forced to be compressed to IQ4_NL, the overall generation quality of the model is not very promising. Given that orcarouter/Qwen3.8-Flash-Next-Uncensored provides the safetensors directly, and we have no other means to obtain the same repack structure (and quality arm, etc.) as the official one, the current BYOGGUF results are quite poor. It would be great if broader support for per-layer quantization levels could be provided... or if the official team could produce a high-quality hgn file!

Since per_layer_token_embd.weight is forced to be compressed to IQ4_NL, the overall generation quality of the model is not very promising. Given that orcarouter/Qwen3.8-Flash-Next-Uncensored provides the safetensors directly, and we have no other means to obtain the same repack structure (and quality arm, etc.) as the official one, the current BYOGGUF results are quite poor. It would be great if broader support for per-layer quantization levels could be provided... or if the official team could produce a high-quality hgn file!

Yes, they could, orcarouter/Qwen3.8-Flash-Next-Uncensored

I made one after halogen-flash-server was upgraded to 0.14:
https://huggingface.co/tatianyi/Qwen3.8-Flash-Next-Uncensored-Halogen