Qwen 27B Q4 at usable speed on 16GB VRAM
Qwen 27B Q4 at usable speed on 16GB VRAM
www.reddit.com /r/LocalLLM/comments/1uxg871/qwen_36_27b_q4_96k_context_running_on_16gb_vram/
Not my settings! All credits to "Stainless-Bacon". I did however replicate the setup just now and I think it's worthy of spreading.
The trick is in a targeted offloading of only a specific kind of layers to CPU making it possible to run a higher quant that otherwise would be too slow to be usable.
Remember to compile llama.cpp with GGML_CUDA_FA_ALL_QUANTS=ON
Original settings below. Since I'm using the iGPU for the system I'm slightly tweaking them to use even more VRAM. On my 5060Ti I'm getting prefill 500-600tps (ub at 2048) and tg at ~10tps.
export GGML_CUDA_DISABLE_GRAPHS=1
llama-server \
--model Qwen3.6-27B-Q4_K_M_MTP.gguf \
--chat-template-file froggeric_fix.jinja \
--chat-template-kwargs '{"preserve_thinking": true}' \
--jinja \
--spec-type draft-mtp \
--spec-draft-n-max 2 \
--fit off \
--n-gpu-layers 99 \
--override-tensor 'blk\.(2[0-9]|3[0-9]|4[0-3])\.ffn_.*=CPU' \
--ctx-size 96000 \
--batch-size 512 \
--ubatch-size 512 \
--cache-type-k q5_0 \
--cache-type-v q4_1 \
--parallel 1 \
--temp 0.60 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--flash-attn on \
--no-mmap \
--host 0.0.0.0