Llama-server 16GB VRAM Configs

This document provides llama-server configurations for a system with an RTX 5060Ti (16GB VRAM) and 96GB DDR5 RAM. The system uses the iGPU, so the full 16GB VRAM is available for llama-server. You can set GGML_CUDA_DISABLE_GRAPHS=1 to prevent out-of-memory errors. Setting GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows small VRAM overshoots without performance loss. This setting causes the system to report the full VRAM size as RAM in use.

Coding and General Tasks

This configuration uses a specialized speculative decoding setup with the llama.cpp-adaptive-kv-streaming fork. This fork supports adaptive KV streaming to improve performance. It uses draft-dflash speculative decoding. DFlash2 improves speculative decoding efficiency.

[Qwen3.8-27B]
chat-template-file = chat_template_qwen3.8.jinja
# Created from https://huggingface.co/byteshape/Qwen3.8-27B-GGUF
# using the tools at https://github.com/bsaleh03/ASCII-Condensed-prune-tools
m = Qwen3.8-27B-ASCII-Condensed-IQ4_XS-3.84bpw.gguf
device-draft = CUDA0
n-gpu-layers-draft = all
md = Qwen3.8-27B-ASCII-Condensed-DFlash2-Q2_K_S-MIX.gguf
spec-type = draft-dflash
spec-draft-n-max = 5
ctx-size = 160000
n-gpu-layers = 99
batch-size = 256
ubatch-size = 256
cache-type-k = q8_0
cache-type-v = q4_0
kv-stream-arena-mib = 4352
kv-stream-spec-keep-pages = 334
kv-stream-spec-reenable-pages = 8
kv-stream-spec-stable-decodes = 4
fit = off
parallel = 1
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
reasoning-preserve = on
no-mmproj-offload = on
mmproj = Qwen3.8-mmproj-BF16.gguf
load-mode = none
flash-attn = on
threads = 16
threads-batch = 16

Model: Qwen3.8-27B-ASCII-Condensed-IQ4_XS-3.84bpw.gguf
Draft: Qwen3.8-27B-ASCII-Condensed-DFlash2-Q2_K_S-MIX.gguf
Fork: https://github.com/troed/llama.cpp-adaptive-kv-streaming
Performance: PP ~600-900 tps, TG ~15-50 tps.

Reasoning and High-Quality MoE

This Mixture-of-Experts (MoE) model provides high-quality grunt work. It uses draft-mtp and ngram-mod speculative decoding. Multi-token prediction (MTP) improves generation speed by predicting multiple tokens at once.

[Ornith-1.5-35B-A3B-APEX-MTP-Quality]
chat-template-file = chat_template_Ornith-1.5-35B-A3B-APEX-MTP-Quality.jinja
alias = workhorse
ctx-size = 140000
flash-attn = on
ub = 2048
fit = off
n-cpu-moe = 23
ctk = q8_0
ctv = q8_0
np = 1
mlock = on
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 2
spec-draft-type-k = q8_0
spec-draft-type-v = q8_0
no-mmproj-offload = on
mmproj = mmproj-Ornith-1.5-35B-A3B-APEX-MTP-Quality.gguf
no-warmup = on
reasoning = on
reasoning-preserve = on
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
load-mode = none

Model: Ornith-1.5-35B-A3B-APEX-MTP-Quality.gguf
Performance: PP ~1100-1500 tps, TG ~25-30 tps.

Prose

This model generates high-quality prose. It uses draft-mtp and ngram-mod speculative decoding.

[gemma-4-26B-A4B]
m = gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
mm = gemma4-26-mmproj-BF16.gguf
md = mtp-gemma-4-26B-A4B-it.gguf
ctx-size = 140000
fit = 0
np = 1
load-mode = none
ub = 2048
n-cpu-moe = 20
spec-type = draft-mtp,ngram-mod
spec-draft-type-k = q8_0
spec-draft-type-v = q8_0
spec-draft-n-max = 4
chat-template-file = chat_template_gemma4_26b.jinja
reasoning = on
reasoning-preserve = on
temp = 1.0
top-p = 0.95
top-k = 64
repeat-penalty = 1.0
jinja = on

Model: gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
Performance: PP ~2000 tps, drops to 1000 tps as context fills.