Developers Pushing Large Language Model Limits
Developers are experimenting with DeepSeek V4 to understand its capabilities, cost, and hardware requirements. They're focused on maximizing performance with limited resources, exploring context window lengths, and seeking ways to distill the model for broader accessibility. The core pain is the complexity and resource intensity of deploying and optimizing these advanced models.
SOURCES (60)
“Ad long as I don't look at the receipt, im always happy with my dsv4 cluster . Its always loaded and ready, just slow at times”
“That was the "Preview" version, as per what they have been saying since April. The next release is basically the official version, or some call it the GA (General Availability) version. And improvements to the model is expected.”
“Deepseek V4 is about to be released; they say it will be cheap, and from the videos I've seen, it will be similar to Kimi K3 and Fable. I think Fable will be replaced by these two Chinese models. submitted by /u/Illustrious-Swim9663 [link] [comments]”
“I bought the 2 M5 Bosgame for £1500 each a while back. I can assure you they are worth every penny. Not the fastest but they do their job and offer 100% privacy for certain things like household + household infrastructure management. $100? no. a tad more. This is just the small LLM related part. I have a bit more hardware than this.”
“I would be *very* surprised if they did not invest in inference optimization. I'm NOT saying they are not investing in optimization. They are. But they are not designing their models to be high value per dollar. They have much larger userbase Larger userbase allows for better optimization as LLM scales better with volume. So, this point is invalid. I've talked to guys from Google Google is the exception amongst the three labs. See Google I/O 2025 , they claim the Pareto frontier for cost”
“Anyone using Terminus-2 as an automonous local agent? This sounds good if properly sandboxed: Terminus-2 uses a unique single-tool approach - an interactive tmux session And from what I gather it is also aware that it's running on docker, right? https://www.harborframework.com/docs/agents/terminus-2”
“TL;DR: I ran DeepSeek-V4-Flash through the same 89-task Terminal-Bench 2.1 suite on two very different local setups: an aggressively quantized 80.8 GiB GGUF on one 128 GB M5 Max MacBook; the native mixed FP8/FP4 checkpoint with DSpark speculative decoding on 2× DGX Spark. The Mac finished at 47/87 graded tasks (54%) . The two-Spark setup finished at 45/86 (52%) . On the 86 tasks graded by both, they agreed on 66 and split the remaining 20 almost evenly: 11 wins for the Mac build, 9 for the Spark”
“Hey all! Wanted to see how DeepSeek V4 Flash GGUFs in two different quants perform on my hardware and share the results. Tested two quants on the fairydreaming/llama.cpp dsv4 fork . As a bonus, I also ran the same model (IQ3_XXS-AS) on mainline llama.cpp b10064 just to check. TLDR: turns out the fork is pointless now! Model DeepSeek V4 Flash made by bullerwins : DeepSeek-V4-Flash.IQ3_XXS-AS.gguf DeepSeek-V4-Flash.IQ2_S.gguf Hardware Component Spec CPU AMD Ryzen 5950X (16C/32T, 14 threads used) R”
“After the recent llama.cpp changes, DeepSeek V4 Flash has become much more usable. I ran some benchmarks and wanted to share the results along with the config I used. I'm using DeepSeek-V4-Flash-UD-Q8_K_XL from Unsloth: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF Config: llama-server \ -m DeepSeek-V4-Flash-UD-Q8_K_XL.gguf \ --override-tensor "blk\.[0-1]\.ffn_(up|down|gate)_exps\.weight=CUDA0,blk\.2\.ffn_(down)_exps\.weight=CUDA0" \ --ctx-size 1048576 \ --cache-type-k q8_0”
You probably want a P5 node from AWS which is about 300K or something
“Hey, it's my first time posting here and I thought I'd share my progress on getting Antirez's imatrix Q2 DeepSeek V4 Flash GGUF (86.7 GB) running on my build. I used this llama.cpp fork which fixed the model's output when KV cache is quantised to Q8. Specs: - Ryzen 7 7800X3D - Asus ProArt B850-CREATOR WIFI NEO - 64GB DDR5-5200 - 2x RTX 3080 20GB (both running at PCIe 4.0 x8, power limited at 280W) Running the Nvidia 610.43.02 drivers on Debian 13. Using these launch parameters: .”
“This is an an insane budget box that I've been using to test out a 98GB model using cpu generation on a 6 core CPU, 16gb vram.. for science. This week it went from 2t/s -> 7t/s on DeepSeek-V4-Flash-UD-Q2_K_XL, which has a 98GB vram requirement . Somewhere between b9986 and b10034 the llamacpp guys are cooking. [tequila ~]$ echo "=== Tequila Hardware ===" && \ echo "CPU: $(lscpu | grep 'Model name' | awk -F: '{print $2}' | xargs) ($(nproc) cores)"”
“It's free on OpenRouter until July 21st. (So happy I just checked, I thought it was only until tomorrow and had neglected to try it. Seeing these results with a lobotomized version? Oh yeah, I'm gonna go have some fun.. And not sleep for a week..)”
“Yes, and I had edited my comment 10 minutes before your previous response to clarify the criticisms. I misunderstood the quantization from a quick google, but the main criticism was the absence of any demonstration of technical understanding in the post, and the apparent claim that there was no tradeoff in accuracy. Calibrating on the test set makes that conclusion hard to defend. To me, this post also feels like a thinly disguised ad for Hey Neo, and I’m sick of companies trying to push claims”
“mimo 2.5 or deepseek v4 flash depending on the task, if you asked about coding i'd say mimo but idk about data management you'd need to personally test”
“Credit goes to: kdzzzds on Bilibili Someone got access to a new version of DeepSeek V4 through A/B testing, and vibe coded / one-shot this No Man's Sky and Minecraft hybrid. Here is the chatlog: https://opncd.ai/share/fnOGJyIn And the source code: https://archive.org/details/no_mans_minecraft submitted by /u/tonyunreal [link] [comments]”
“I've been using DeepSeek 4 flash, it's really good. Pat as you go pricing, but it's so cheap that it's less than other subs.”
“Oh, is it faster now? I'll have to check it out again. I was using a few of the PRs out there and even though the model is way smaller, it was running way slower for me than even bigger ones like GLM 5.2.”
“DS4 is much faster on llamacpp with the last couple of unmerged PRs applied. And, it seems to have less problems with reasoning loops and broken tool calls, though that could also just be because it's small enough to run at q8 compared to the lower quant I was using with Mimo.”
“Try --no-kv-offload . No seriously. I have a 7900XTX and DDR4, and I'm getting 120-150 pp (same as you) and 6-8 t/s decode, with a Q3XXS . Without -nkvo, my decode speed is 3 t/s. You have much more compute and way higher bandwidth, yours should be screaming. With models this huge it's cheaper to ship context through PCIe rather than activation vectors, so keeping it in system RAM makes things faster, counterintuitively.”
“Disclosure: No AI was used to write this My specs are: RTX 4090 128 GB DDR5 5600 MT/s Intel Core Ultra 7 270k Running nvidia-595 on ubuntu 26.04 with latest llama.cpp build (pulled and rebuilt this morning). Tried a lot of things, ended up running unsloth's UD-Q2_K_XL quant with command: taskset -c 0-7 /home/kevin/ai/llama.cpp/build/bin/llama-server -lv 4 -m /home/kevin/ai/models/DeepSeek-V4-Flash-UD-Q2_K_XL-00001-of-00003.gguf --temp 1.0 --top-p 1.0 --min-p 0.0 -t 8 -fitc 64000 -fa off -np”
“Both deepseek ocr and dots ocr perform worse than glm ocr on omnidocbench: https://github.com/opendatalab/OmniDocBench”
“When I recently built my inference server, I expected to deploy DeepSeek v4 flash, but that doesn't look like it's going to be fast for a long time, if ever. There is a massive gap, as we all know, in competent models between 30b and 400b. I was very surprised to find that this is the best model by far that falls within this gap. MiMo v2.5 the only local model I have seen that's faster than fast cloud providers. It is actually worth it to build a big server just to run this thing. Ru”
“Figured some people might want to watch while I tinker in real time. https://youtube.com/live/K3msQYd-vn0?feature=share submitted by /u/GoodMacAuth [link] [comments]”
“Obviously, I did. But nevermind, I actually figured out what my issue ws the other day and it's working in llama.cpp now. :) I had an embeddings server launching with it that was eating into my ram, causing pp to crawl. Moved it to cpu and also have option to load w/out it. Anyway, works fine now. Getting 45 tok/sec steady, sometimes peaking over 50.”
“Trust me. Anthropic and OpenAI usually release new models around this time. submitted by /u/Original_Judgment494 [link] [comments]”
“I've been running DeepSeek as my daily driver. It's brilliant at reasoning, architecture, debugging... everything except: it can't see images. Every time I hit a visual problem (an error dialog, a UI mockup, a chart) I had to break flow, upload the screenshot to Gemini, ask it to describe what's on screen, then paste the description back. Kills my workflow. Also means my screen is on somone elses servers. So I built LocalEyes . A Claude Code skill that gives DeepSeek working eyes”
“In my opinion this is not true knowledge distillation but rather standard finetuning on synthetic data. 1200 examples are only enough to teach the model a specific format or style, rather than transferring the teacher's actual capabilities.”
“I have nothing against him learning, but naming this distillation is the same as turn boiling liquid mixture off right away as it only began to boiling not waiting any separation. Because of this naming we have a bunch of useless models with distillation in name. 5% distill Alcohol?”
“It's young... let's wait for some days/weeks to see the real improvements appear”
“So I decided to learn how to fine-tune LLMs. Read a few guides from Unsloth, poked around, then stumbled on Unsloth Studio and wanted to test it out. The dataset I started from a set of relatively unrelated QA pairs — Natural Questions — and stripped the answers. Then I had DeepSeek v4 Pro (thinking disabled) repopulate them: - 1000 train + 200 val = 1200 requests total, cost $0.36 (~$0.0003/req). Honestly impressive on DeepSeek's side. Unsloth Studio It's a huge pain in the butt — infes”
“Hey all, I've been working with DSV4 on my Frankenstein rig (5-GPU box: 2x3090 + 5060 Ti + 2x4060 Ti, 96 GiB VRAM, 125 GiB DDR4) doing agentic coding with pi and ran into some serious issues. After much work I submitted an issue and have a fork with fixes. I would love for maintainers and DSV4 users to reproduce and try the fix. My user experience is much better, even at 10 t/s. This is deterministic and jams at iteration 29 of every run. The problem is on upstream master. My fix forks from”
“I run the q2 while doing other stuff. Typically around 15 tok/s gen, 200 tok/s prompt. I reduced batch size to free precious GB, as it is a tight fit.”
good point! i do these as a hobby lately https://llama.garden/
“MiMo V2.5 is 310B A15B. DS4 Flash is 284B A13B. Neither has 21B active parameters.”
“MiMo-V2.5 is better. DSv4 Flash just seems to have a cult-like status because it is from the mythical DeepSeek and people wasted a bunch of effort developing custom inference engines to run it. Sunk cost fallacy: they decided to keep claiming it was amazing everywhere. But, DeepSeek is supposed to be releasing the fully trained version of DS4 Pro/Flash in a few weeks, and that might surpass MiMo-V2.5. Hopefully the non-preview release of DS4 will also include vision. DeepSeek has teased that it”
“Hello guys. Sharing my experience with deploying DS-4-Falsh with DSpark on HGX-H200 For context of my setups etc (including how the hell i have access to H200), you can read my previous posts So mostly, its about my comparison of marlin moe_backend, that i used on my previous main deploy config with EAGLE (1-1-2), and flashinfer i saw in official blog ( https://www.lmsys.org/blog/2026-07-06-dspark-sglang ). In general i can say, that DSpark absolutely faster then EAGLE . Didnt save exact tables”
“I noticed while using both, mimo was often better, after benchmarking mimo v2.5 via open code endpoint in diff harness like codex, oh my pi, hermes. i found that mimo is indeed better in coding tasks. and over all, hermes scored 55% with mimo v2.5 via terminal bench v2.0 others did under 50% too with any harness from my list or deepseek v4 flash Not that i dont like deep seek v4 flash its GOAT, i have used it more. but as per benchmark both models are same at most places but when u run real life”
“Pipeline parallelism overlaps ubatches inside the batch across pipeline stages. Corollary there is you actually have to have a reasonable number of ubatches per batch or you get pipeline bubbles. With 8 GPUs, you probably want like -b 8192 -ub 512 or something.”
“Trying now w/ openrouter. Actually a good test, I’m debating a Third dgx spark. I got glm into 2 but it was at 1bit and 4k context, and any prompt hard locked the cluster.”
“I'm content that if tomorrow all open weight releases stopped, true...but like, official deepseeks this month and if we get gains similar to hy3 preview to official...it's truly peak local LLM”
“it says Q4 is only 7gb less than q8 (162GB) - how? Also anyone know which version would work for 120gb/128gb and if it'd run on koboldcpp?”
“Oh yeah, it can't run dense models well at all due to the limited bandwidth, but the advantage of Framework is when I needed to I could just swap in the added RAM i needed. When I started getting into local LLMs last year I just swapped in 96gb of RAM. Then when I decided I wanted faster Whisper.cpp transcription and embedding models for RAG locally, I picked up the 7700s :)”
“I’m on an M3 Ultra studio 256GB. I normally run ds4.c (the Q4 experts quant). I tried the Q8 unsloth quant with the recommended MTP PR and was disappointed. For some reason it occupied almost all of my memory at 128k context, even with context quantized to Q8, while ds4.c takes up about 180GB at the same context length. Then I tried a very short prompt, and decode was about 6 tokens per second (versus 23 tokens per second on ds4.c). I am now back on ds4.c, needless to say.”
“Thanks for the notes and u/tarruda for the feedback, I did, in fact, go a bit lazy-mode with the template. The corrected template PR is here: https://github.com/ggml-org/llama.cpp/pull/25414”
“Basically the title. When I tried to run it I got a lot of problems. On my old Xeon rig it was too slow. With llama-server the follow up questions resulted in a mess, as neither initial question nor reasoning was fed back to the model. When I tried to run it with latest stable LM Studio there were no way to make model to think before answer. So please state your hardware, present cli command to run specifying quant version and report tg and pp speeds. If you experience any problems, please, stat”
“You really skimmed the part about needing the vram to fit the whole model in RAM, huh? You’re still reserving 35gb vram vs 27gb.”
“That link goes to a zhipu ai article. As far as I know deepseek hasn't trained any model on huawei chips, only inference.”
“Got my Ascent GX10 two days ago and spent the last couple of days pushing a 162B REAP-pruned NVFP4 DeepSeek-V4-Flash (a 284B active 13B model) setup on a single Spark by patching the eugr/spark-vllm-docker image. Credit where it’s due: the REAPs were done by 0xSero . I’m just the person who wired it up, validated it, and pushed it through the machine. The main thing I wanted to check was long-context consistency, and the interesting part is how steady the throughput stays as context scales up. I”
“Hi. I don't have a GPU. So my biggest "local LLM" experience has been running ~26B models with single-digits tps values. However, the "serving economy" of DSv4 models look like a riddle to me. The Flash model has 284B parameters, but providers (e.g. OpenRouter) charge so little for it it's ridiculous. It's for example cheaper than 27B Qwen, A tenth of its (total) size! How is it viable? Are the providers just doing dumping here? Or is DSv4 architecture somehow dif”
“Hey guys, just wanted to share my recent homelab project. I used a reComputer RK3576 board, flashed the new fnOS 1.1.31, and expanded it with a 128GB SSD. Managed to bridge the local storage directly to the Lark API using Nanobot. Now I can fetch/upload files via chat client anywhere with zero port-forwarding. Response time is around 3-7s. Check out the setup and the test demo here: No Port Forwarding! Turning Lark/Feishu into a NAS Client via fnOS & RK3576 - YouTube submitted by”