Llama-server 16GB VRAM Configs

This page documents llama-server presets for one system. The system has an RTX 5060 Ti with 16GB of VRAM and 96GB of DDR5 memory. The display output runs on the iGPU. This leaves the full 16GB of VRAM free for llama-server.

Set two environment variables for the best results. Set GGML_CUDA_DISABLE_GRAPHS=1 if spurious out-of-memory errors occur. Set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 to allow a small VRAM overshoot. This keeps the speed the same. The system then reports the full VRAM size as RAM in use. Other running programs may be affected.

Each preset is an ini section for llama-server. Copy the section into your models-preset.ini. Replace the filenames with your local paths.

Coding and Long Context

Qwen3.8-27B is a hybrid model. It has 16 full-attention layers and 48 recurrent blocks. The model file prunes the byte-level BPE vocabulary from 248,320 rows to 129,006 rows. It keeps 256 byte-fallback tokens. The freed VRAM becomes KV cache. The model comes from byteshape/Qwen3.8-27B-GGUF . ASCII-Condensed-prune-tools performs the prune. The tool gathers rows in quantized space. The output is bit-identical with a direct quant of the pruned model. The preset sets a 200,192-token context window.

This preset needs troed’s fork of llama.cpp . The fork extends Raymond’s adaptive-KV streaming work . The fork keeps the full KV history in pinned system RAM. It holds only a bounded working set on the GPU. A bounded working set fits a 200K context in 16GB. A PCIe 5.0 link keeps the streaming fast. Build the fork with -DGGML_CUDA_FA_QUANTS=q8_0-q4_0,q8_0-q8_0,q4_0-q4_0 for the KV cache quants in use.

The preset runs two speculative methods at once. The model embeds an MTP head. It drafts up to five tokens per step. An ngram drafter proposes wide token rounds. spec-draft-sampling = probabilistic samples the draft tokens. The target verifies them with rejection sampling. spec-draft-p-min = 0.7 ends a draft round early when a draft token scores below 0.7. shared-device-memory-mib is an alias of --kv-stream-arena-mib. It sets the GPU arena budget. The arena holds the compute workspace and the streamed KV grants. A DFlash2 draft file exists for standard llama.cpp. The fork’s arena supports MTP, not DFlash2.

[Qwen3.8-27B]
log-file = llama-server.log
chat-template-file = chat_template_qwen3.8.jinja
# Created from https://huggingface.co/byteshape/Qwen3.8-27B-GGUF
# using the tools at https://github.com/bsaleh03/ASCII-Condensed-prune-tools
m = Qwen3.8-27B-ASCII-Condensed-IQ4_XS-3.84bpw.gguf
spec-type = ngram-simple,draft-mtp
spec-draft-sampling = probabilistic
spec-draft-n-max = 5
spec-draft-p-min = 0.7
device-draft = CUDA0
n-gpu-layers-draft = all
ctx-size = 200192
n-gpu-layers = 99
batch-size = 256
ubatch-size = 256
cache-type-k = q8_0
cache-type-v = q4_0
shared-device-memory-mib = 4096
fit = off
parallel = 1
temp = 1.0
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
reasoning = on
reasoning-preserve = on
mmproj = Qwen3.8-mmproj-BF16.gguf
# mmap slows down PP
load-mode = none
flash-attn = on

Model: Qwen3.8-27B-ASCII-Condensed-IQ4_XS-3.84bpw.gguf
DFlash2 draft: Qwen3.8-27B-ASCII-Condensed-DFlash2-Q2_K_S-MIX.gguf
mmproj: mmproj-bf16.gguf
Prune tools: bsaleh03/ASCII-Condensed-prune-tools
Engine: troed/llama.cpp-adaptive-kv-streaming
Performance: PP ~500-1000 t/s, TG ~15-70 t/s on the 5060 Ti.

Prose

Gemma 4 26B A4B is a Google Mixture-of-Experts model. It has 25.2B total and 3.8B active parameters. It has 30 layers and a 256K context. The preset targets prose, not code.

The model is 23GB at Q6_K. It does not fit in 16GB of VRAM. The preset runs on standard llama.cpp. n-cpu-moe = 21 moves the MoE expert weights of the first 21 layers to the CPU. Attention stays on the GPU. The 96GB of RAM holds the offloaded experts.

The preset runs two speculative methods at once. MTP (Multi-Token Prediction) reads a separate draft file. It predicts the next few tokens; the target checks them. ngram-mod needs no draft model. It keeps a small hash pool of n-grams from the context. It proposes continuations that already appear in the text. The two methods run independently.

[gemma-4-26B-A4B]
m = gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
mm = gemma4-26-mmproj-BF16.gguf
md = mtp-gemma-4-26B-A4B-it.gguf
ctx-size = 140000
fit = 0
np = 1
load-mode = none
ub = 2048
n-cpu-moe = 21
spec-type = draft-mtp,ngram-mod
spec-draft-type-k = q8_0
spec-draft-type-v = q8_0
spec-draft-n-max = 4
chat-template-file = chat_template_gemma4_26b.jinja
reasoning = on
reasoning-preserve = on
temp = 1.0
top-p = 0.95
top-k = 64
repeat-penalty = 1.0
jinja = on

Model: gemma-4-26B-A4B-it-UD-Q6_K_XL.gguf
MTP draft: mtp-gemma-4-26B-A4B-it.gguf
mmproj: mmproj-BF16.gguf
Engine: standard llama.cpp
Performance: PP ~2000 t/s, drops to ~1000 as the context fills.

Agentic Tasks and Video

Nemotron 3.5 Bolt is an NVIDIA Mixture-of-Experts vision model. It has 74B total and 9B active parameters. It mixes Mamba-2 and Transformer layers. It reads text, images, and video. The preset targets agentic tasks and video.

The IQ2_XXS quant is 22GB. It does not fit in 16GB of VRAM. fit = 1 lets llama-server adjust unset arguments to fit the device. fitt = 800 keeps an 800 MiB margin per device. moe-cache-mib = 2048 gives the GPU a 2048 MiB LRU cache. The cache holds the most recently used MoE experts that live on the CPU. cpu-mask = 0xF00 and cpu-strict = 1 pin the CPU work to four cores.

The model has no MTP layers. The draft file is a LiLiCorr drafter with 1.13B parameters. NVIDIA ships it with the model. DFlash2 adds grouped dynamic depthwise convolutions to DFlash. A candidate selector picks the tokens for each draft round. Standard llama.cpp merged DFlash2 support in PR 27342 . Set spec-type = draft-dflash to use the draft. troed’s fork also runs the draft. The fork fixes a graph-workspace problem for Nemotron 3.5 Bolt.

Do not add an ngram drafter to this preset. Draft tokens can pull CPU experts into the GPU cache during verification.

[nvidia-nemotron-3.5-bolt]
m = NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ2_XXS.gguf
mm = NVIDIA-Nemotron-3.5-Bolt-mmproj-BF16.gguf
md = dflash-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4.gguf
ctx-size = 160000
batch-size = 4096
ubatch-size = 1024
cache-type-k = q8_0
cache-type-v = q4_0
fit = 1
fitt = 800
moe-cache-mib = 2048
load-mode = none
np = 1
# don't add an ngram drafter - risk pulling in experts for verification
spec-type = draft-dflash
spec-draft-n-max = 6
spec-draft-type-k = q8_0
spec-draft-type-v = q4_0
reasoning = on
reasoning-preserve = on
temp = 1.0
top-p = 0.95
top-k = 64
repeat-penalty = 1.0
jinja = on
cpu-mask = 0xF00
cpu-strict = 1
threads = 4
threads-batch = 2

Model: NVIDIA-Nemotron-3.5-Bolt-74B-A9B-IQ2_XXS.gguf
DFlash2 draft: dflash-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4.gguf
Official drafter: nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF
mmproj: mmproj-BF16.gguf
Engine: standard llama.cpp or troed’s fork
Performance: PP ~700 t/s, TG ~50 t/s on the 5060 Ti.