Looks like DS4 runs at about the same inference speed as llama.cpp when using ROCm, but has about 40%-50% faster prefill than llama.cpp. I've been making tweaks to open up the context to 250k, currently looking into the impact of trying to go for 393216 tokens, which is the think-max value, but my 5GB KV cache will expand a bit, bringing me too close to my 96GB limit. Will report back. To get it running the first time, I used Opencode-hosted Deepseek v4 0731 (or maybe 4.1, not sure), and it ended up modifying the source code to avoid a dependency on HIP (host+device in CUDA):

A genuine DS4 source issue — rsqrtf() is device-only in
HIP (host+device in CUDA) — patched 
rocm/ds4_rocm_deepseek4_vision.cuh:239 to 1.0f/sqrtf(). 
That's a local uncommitted patch worth sending upstream; 
the kyuz0 fork only compiles it on ROCm 10.0.