Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of the performance from dual 3060 with RAM offloading.
DeepSeek V4 Flash 0731 IQ2_M from Unsloth
CPU: Ryzen 7500F GPU 0 (PCIe 5.0x16 lane): RTX3060 GPU 1 (PCIe 3.0x1 lane): RTX3060 RAM: 96GB 5600
Prompt: Write me a tetris game.
Result (copy from the summary): Prompt eval: 2.96s Prompt speed: 3.0 tok/s Generation: 960.02s Speed: 4.5 tok/s (PowerShell shows 3.5 tok/s, don't know why it shows 4.5 tok/s, single GPU with RAM offload output was around 3 tok/s so I trust PowerShell metrics more) Tokens: 4,338 First token: 2.96s Cache hits: 1 Total: 963.32s Chunks: 4318
Wattmeter is on the way but my estimate is around 130W total system power draw (with 2 monitors connected but they are not taken into account) Each GPU used 30-40W with 0.9V undervolt. Task took around 16 minutes to complete, consumed around 35 watts and cost 0.0059 euros.
i got around 10 tok/s with 2x RTX 3060 128GB ddr4 cpu amd 5950x DeepSeek-V4-Flash-0731-UD-IQ3\_XXS
Forgot to mention, everything is on Windows 11. How did you got 10 tok/s? I thought it is 2-3 RTX3090 + RAM territory.
can you share your settings? This is significantly faster than I would have guessed.
Wonder if your bottleneck is actually cpu
:\]
You can do tetris easy that with Qwen3.6...
Or Gemma 31B for that matter
It was more a speed test than a skill test.
I get around 3.5tps on a MacBook Pro M5 Pro with 48GB of memory….
Have you tried Dwarf star yet? It may double you up. It's meant for 96GB+ though so not sure if it would work or not.
I have 2x 3090 + 64GB DDR5 with a low-end AMD CPU from 8000 series. UD-Q2_K_X gives me around 100 tps prefill and gen is pretty stable at 15. A bit rough, but much better than I expected actually…
This is much faster than me. 2x3090 and 64gb DDR5 ECC 4 Channel on a xeon W5 running 8.2tk/s for flash game coding (Unsloth). Are you on Llama.CPP?
I feel your pain. This is quite generous: https://huggingface.co/spaces/victor/DeepSeek-V4-Flash-0731-free-endpoint
You serious. That's your test? Build a tetris game ... Qwen 27b 3.6 do it easily and much more. DS 4 lite should build drivers , build full applications , improving Vulkan , cuda kernel for itself, make full 2d, 3d games but you are using that model to build Tetris ... the game which local models were doing that in 2024.
Aside from being easy it's a *"solved"* problem that's likely verbatim in training sets. Everyone should have a set of weird/unique-to-them tests to run against new models. 2 years ago this manifested as models being unusable for code-editing but every single one of them could make a perfectly functional flappy-bird game lol
3 tokens per second prompt processing is brutal
This was small prompt, haven't tried to analyze anything big yet. Just few prompts that came to mind first. Bigger one gave around 70 tok/s processing.
I wouldn't be surprised if your pcie lanes for you second gpu is the huge bottleneck here, since pcie 5 x16 for gpu 1 gives you 64GB/s, and pcie 3 x1 on gpu 2 is only 1GB/s. If you had a threadripper or similar enterprise board with more lanes you'd probably see improvement.
The Q3 seemed just too big for me 96 GB of RAM + 40 GB of VRAM. But now, you are making me give Q2 a try.