`-sm tensor` would make this about 50% faster from what I see. But that's not merged yet.
As a man, I'm ashamed to admit that these pp results are not great. I'm running IQ3_XXS in full VRAM over 5 GPUs (2x 3080, 2x 3090 , 5090), and the speeds are not great. As stated in the title, ~600 t/s pp; that's not usable as a daily model for me. Even running UD Q4 K XL I can get 200-400 PP t/s with partial offload. I verified the model is fully in VRAM. I did --fit and normal layer assignment. This is also with the latest CUDA, NCCL, and Llamacpp builds. I'm not familiar with profiling LLamacpp but I am looking into that. While I do that, can others chime in on their PP numbers?
`-sm tensor` would make this about 50% faster from what I see. But that's not merged yet.
Try dwarf star for ds4. Current Lamacpp is not very well optimized yet. https://github.com/antirez/ds4
It would be useful if you could share your full config, so we can all compare. I'm getting around 500 t/s pp, 35 t/s tg at empty context with the following config over 8 RTX 3090. Context comes out at 200k. The gguf is the ggml one. With the Unsloth Q8, I get around 450 t/s pp, 32 t/s tg. llama-server --host 0.0.0.0 --port 8080 --flash-attn on --fit-target 1024 --seed 3407 --prio 3 --parallel 1 --log-verbosity 4 --split-mode layer --threads 24 --threads-batch 48 --main-gpu 0 --no-mmap --direct-io --jinja --model /storage/gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf --chat-template-file /home/hubby/llama.cpp/models/templates/deepseek-ai-DeepSeek-V4.jinja --fit on --temp 1.0 --top-p 0.95 --min-p 0.0 --batch-size 4096 --ubatch-size 4096
This model architecture is a lot different from most other models and trades high PP compute complexity for very low KV cache use per token. It’s not really fair to compare its PP speed against other models when it can fit 1M context in like 1/4 the KV cache equivalent size of GDN/SWA models and 1/16 of full attention models. OTOH if it’s too slow to be useful for you then sure that’s a problem. Hopefully we’ll see some continued improvements from kernel optimizations.
no, it doesn't. DS4F is known for low computation compare to many other attention such as KDA and Qwen3next especially at very long context. The problem is there is no proper implementation for kernels.
Depends on what you compare to. Opus, Fable and Gpt were horribly slow until last week or so (they magically recovered after K3 joined the market) So two weeks ago I would have taken your values over the big three closed models. Now they are ok, let's see for how long.
\-b and -ub params affect prefill speed. Increase them (try 8192) if you have enough space.
Better performance than im getting on the Q8 version. 25 ts and \~300-400 pp
I'm ashamed to admit that these pp results are not great.
"Doesn't have to be great. Just good." - what she said