`-sm tensor` would make this about 50% faster from what I see. But that's not merged yet.
Does sn tensor work with a setup like a pro 6000 Blackwell + 3090 ?
As a man, I'm ashamed to admit that these pp results are not great. I'm running IQ3_XXS in full VRAM over 5 GPUs (2x 3080, 2x 3090 , 5090), and the speeds are not great. As stated in the title, ~600 t/s pp; that's not usable as a daily model for me. Even running UD Q4 K XL I can get 200-400 PP t/s with partial offload. I verified the model is fully in VRAM. I did --fit and normal layer assignment. This is also with the latest CUDA, NCCL, and Llamacpp builds. I'm not familiar with profiling LLamacpp but I am looking into that. While I do that, can others chime in on their PP numbers?
`-sm tensor` would make this about 50% faster from what I see. But that's not merged yet.
Does sn tensor work with a setup like a pro 6000 Blackwell + 3090 ?
Try dwarf star for ds4. Current Lamacpp is not very well optimized yet. https://github.com/antirez/ds4
The 1 downvoter has no clue at all
It would be useful if you could share your full config, so we can all compare. I'm getting around 500 t/s pp, 35 t/s tg at empty context with the following config over 8 RTX 3090. Context comes out at 200k. The gguf is the ggml one. With the Unsloth Q8, I get around 450 t/s pp, 32 t/s tg. llama-server --host 0.0.0.0 --port 8080 --flash-attn on --fit-target 1024 --seed 3407 --prio 3 --parallel 1 --log-verbosity 4 --split-mode layer --threads 24 --threads-batch 48 --main-gpu 0 --no-mmap --direct-io --jinja --model /storage/gguf/DeepSeek-V4-Flash-0731-MXFP4.gguf --chat-template-file /home/hubby/llama.cpp/models/templates/deepseek-ai-DeepSeek-V4.jinja --fit on --temp 1.0 --top-p 0.95 --min-p 0.0 --batch-size 4096 --ubatch-size 4096
Oh, that is definitely not what I would expect for an MoE model fully loaded into VRAM... I hope we can get better llama.pp optimization for this model
This model architecture is a lot different from most other models and trades high PP compute complexity for very low KV cache use per token. It’s not really fair to compare its PP speed against other models when it can fit 1M context in like 1/4 the KV cache equivalent size of GDN/SWA models and 1/16 of full attention models. OTOH if it’s too slow to be useful for you then sure that’s a problem. Hopefully we’ll see some continued improvements from kernel optimizations.
no, it doesn't. DS4F is known for low computation compare to many other attention such as KDA and Qwen3next especially at very long context. The problem is there is no proper implementation for kernels.
Depends on what you compare to. Opus, Fable and Gpt were horribly slow until last week or so (they magically recovered after K3 joined the market) So two weeks ago I would have taken your values over the big three closed models. Now they are ok, let's see for how long.
\-b and -ub params affect prefill speed. Increase them (try 8192) if you have enough space.
Better performance than im getting on the Q8 version. 25 ts and \~300-400 pp
I'm ashamed to admit that these pp results are not great.
"Doesn't have to be great. Just good." - what she said
API is still incredibly cheap. I would use it as long we have those prices, no local setup can compete with that
if 600pp is not good enough for you, sell your rig and pack it up. you have got to be kidding me.
No it's not good enough for me. My context starts at 15k with just the prompt so I don't want to wait. Currently, we're dispatching a ton from the CPU and that will eventually be resolved and this model will work much faster.
Downgrade your CUDA. With 13.2 DeviceTopK is used for top-k instead of argsort, this turns PP rate to crap.