AI Benchmark Scores Don't Match Real Task Performance
Users are experiencing a disconnect between benchmark results and real-world performance of AI models, particularly Qwen. They face challenges with operational realities, model stability, bias mitigation, and suitability for specific tasks, despite promising initial scores. This highlights a gap between theoretical performance and practical usability.
SOURCES (60)
“Yeah, they’re obviously going to continue to run post training. These models keep getting better the longer you let them train, and this is only the first checkpoint of a brand new base model. From the benchmarks they shared, it looks like they decided to…”
“Mach-1-Additive-35B-A3B too. Problem is delay on GGUFs & PRs on mainline. Model creators should spend little bit time on PRs & GGUFs. They created models to reach more people, right? So PRs & GGUFs are fastest way. I'm tracking these models on below thread 1-bit / 2-bit / Ternary / Bitnet Models - Updates & Tracking”
“that repo's langchain's, not langfuse, fwiw. jacek's thread covers most of the maintained ones but local-deep-research from LearningCircuit didn't come up there. it points at any openai compatible endpoint so vllm or llama.cpp just work. no kiwix or firecrawl retriever in it though, you'd extend BaseSearchEngine and hit kiwix-serve's /search endpoint for previews then /raw/<zim>/content/ for the article body. for the notebooklm side, open-notebook, it's the clos”
“Why not GPT-5.6 Luna if you use cloud models? What benchmarks exactly? I feel like this post is from the middle of conversation somehow”
“I guess we can also assume these labs have some internal benchmarks to judge progress during and after training and QA/testing on top of that so I doubt they release a model which isn't better at least in some aspects.”
“How do you see it ending? Keeping in mind all the chinese models are made open source?”
“ARC-AGI 3 is at like 99.9% .. Humans are the only ones to score 100% on those. Holy crap... ASTRA IS ALIVE. Literally, alive!!”
“Again? Havent they already said that about previous versions? 🤣 But yeah that sub is cringe.”
“Fair but the people qualified to build it said AGI is here..so some credibility to it.”
“getting the same AA score as Sol, makes me wonder how much can we even rely on these benchmarks. We wouldn't know how Astra does in our specific use cases, but then we might have to try to limit the bias we get looking at them. Looking at this chart alone we might interpret astra has close to zero improvement over sol while being 2.5x times more expensive on token price and possibly more efficient on token usage. Id say lets wait until we get our hands dirty.”
“It overcomplicates so much, which I suppose is it's point...but it's been a detriment to my workflow many times. Which begs the question of what specific metrics they're using to make such a statement, because last time I checked, developer productivity and software quality are insanely hard to get good metrics for.”
“Seeing as an estimated 30% of the problems in ExploitBench are mathematically impossible, how exactly do you get 100% without cheating?”
“I don’t notice a difference to be honest and Ive been on Pro ($200/m). I’ve also just gotten in the habit of creating skills for my projects that way 5.6 never veers off track or loses focus to the point that it doesn’t know what to do”
“Everyone has heard of Qwen, Gemma, Muse/Llama, and GLM. Many have heard of Nemotron, MiniMax, Ling, and LFM. Some have heard of Laguna, MiMo, and Inkling. I don't really see any discussion about, say Dots and Voyage Code. That's the level of obscurity I'm curious about. EDIT: Excluding fine-tunes or suspected fine-tunes. A lot of them are good, but I'm curious about foundation-level models that people are sleeping on. I'm aware some of them probably started as fine-tunes. ”
“Just like the one before, and before that, and before that, and before that. According to the headlines we should be far into AGI territory.”
https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra
“OpenSource models like to over-reason on every problem. I put together a notebook and a video implementing grpo from scratch and using it to post-training Qwen 3.5-2B to improve its accuracy and reasoning efficiency. The results were quite interesting, despite training it purely on the task of simulating the python interpreter, the model became a lot more accurate and token efficient on math problems. The code can be applied to any open source model. Here is the code agi-playground/grpo at main”
“https://preview.redd.it/yucrkdn23cnh1.png?width=1104&format=png&auto=webp&s=3d9b3c7e14f600b914ce452c3b2049eb3d767196 WTF They really cooked. https://simple-bench.com/ submitted by /u/Healthy-Nebula-3603 [link] [comments]”
“This is exactly the kind of thing I was hoping people would share here. The 76% vs 44% really says it all — you were improving the step you thought was broken instead of the one the data showed was actually stopping people. I think I may have done a version of that too. I built pretty heavily around a “captcha” style onboarding idea for PayaChat where after creating an account new users might have to do something playful like a push-up or a certain pose of their choice before they can even enter”
“The problem with the down detector main reporting page is that all of the graphs are scaled to the same size. The OpenAI spike was nearly 40,000 and the Google spike was just over 100 (just over 400 for Gemini). They look the same in the reporting page.”
“That linked LinkedIn article cites my earlier post about Thomson, but I have a more-recent one that provides some additional benchmarking info. TR said it would publish a technical report wih more detailed info, but I haven't seen it. https://www.lawnext.com/2026/08/thomson-reuters-launches-thomson-its-own-proprietary-llm-trained-on-westlaw-and-practical-law-content.html”
“After extensive testing of all of these, and frontier models, for any real C++ work spanning hundreds of thousands of lines and hundreds of individual files, especially work which involves complicated realtime rendering (not even vision models help), all of these are 0 useful tokens per second, and human is still 0.5 but requires 30+ years of fine tuning. LLMs are excellent at generating mediocre, uninspired, "expected value" web apps, though. Everything else is just LLM nerds doing si”
“No to mention they are useless for at least 25 years. And the failure rate is so high that most models never amount to anything.”
“This graph is very confusing. The title says "Non-Hallucination Rate" which implies higher is better, but the subtitle says "hallucination rate" which implies lower is better. And since there are small and big models all over the place, I have no clue which one is which.”
“Not everything is about coding. Seems like the "hardcore AI" people forget that most poeople dont use AI for coding, even on the high end. So comparisons like this are kinda wasted on most people, as you use coding as your only criteria.”
“nice, great to see more high end open weight models. Also i tersting to see something like this from a US lab.”
“Yeah, I have seen this behavior before. It is just the random pick where it doesn't parse what you mean on step one. Its not the model per se, it is the behavior of these slot machines. I try to mitigate by checking it at least starts out with what I want for the first turn, and if it goes off script I just stop it.”
“This is finetune.... Maybe I’m missing something, but it seems to me that David inadvertently lobotomized this model when he drastically reduced model's thinking duration. I have no idea where those benchmark scores are coming from I tested it by asking it to create a single-file HTML tower defense game, using both a very simple prompt and a highly detailed one in the DeepSeek harness, and in both cases it performed worse than the stock model and its predecessor (DavidAU/Qwen3.8-27B-Cold-Fus”
“Yes https://llm-stats.com/benchmarks/mrcr-v2-(8-needle) Seems all current models solved it. It was quite bad few moths ago yet.”
“This feels like a really clever release by Qwen. After all of Anthropic's "distillation attacks" complaints, this release really shuts that up considering its (projected) performance is near Fable 5.1 which was just released.”
“Did you ever try to optimize the --override-tensor ? You should get some more ctx out of that.”
Just going to wait and see Artificial Analysis Index. Fable 5.1 got 66
“I mean the moment I saw "gpt-oss" I had a small suspicion, and by the time I read up to the unrendered markdown tables I had the same thoughts as you”
“Well actually Gemma 4 figured a difficult issue for me that Qwen 3.8 could not tackle yesterday. It involved analyzing a 11k lines file so not your everyday task though.”
“You mean LiquidAI’s LFM2.5-Encoder-350M-PII-Detector? I haven’t tried it yet, but it’d be interesting to run it on the same 200 docs.”
throwing my hat into the ring. it's a video or image model.
“Hot take? Who cares? Now, of course, the answer is "at least a few people" do actually care and have tasks that only Fable can take on, but we're talking fraction of a fraction at this point. I'm (like I suspect many here are) a VERY heavy user of AI for all kinds of tasks, business/agentic, coding, scripting. I'd put myself into the top 10% of "advanced" AI users and I can't come close to inventing a "real task" that a large open model can't tak”
“Ah gotcha. Thanks for the info. I will have to go investigate their discord, I didn't even know it existed.”
“English and Chinese are the only important languages. Don't waste the training on multilingual.”
“I rly wonder how Qwen 3.8 27b makes it into these benchmarks - tried several agentic tasks with it and the results are… underwhelming.”
Unlike sonnet and gemini? They publish their training data?
“Will the end user experience sufficient differentiation, and is the model good enough to be used in the first place? That's a different question to whether you believe the research is novel or interesting. The report feels miles away from a commercially viable product, especially given the level of competition in AI.”
“Right. These damn benchmarks need error bars. Especially the stupid Harvey “all pass” metric.”
“I agree that loss of generality is crucial. That is likely why the report evaluates target-domain gains separately from general-capability preservation, measures both against the original Qwen checkpoint, and reports the regressions it found. “Benchmaxxing” is an evaluation risk. It is not evidence that catastrophic forgetting occurred. And no, I do not mean “RLHF on custom tool calls.” The reported pipeline uses a DPO warm start followed by on-policy GSPO over complete trajectories containing r”
“With language models loss of generality is crucial, you cant trivially disambiguate it from domain-specific performance (hence "benchmaxxing"). "Training inside its own tool environments" means nothing, I assume you mean RLHF on custom tool calls- which ones? Your comments read like TR marketing material.”
“Long story short, about a year ago, in spite of everybody bashing gpt-oss for broken tool calling and refusals, I thought there's something there worth exploring. Model hit a sweet spot for me in that it was the first time I could run full 128k context, factory-precision weights, across parallel requests on a single RTX 3090 at close to 200 tps (well... eventually, but it was still flying at around 100 tps initially which was mind blowing in the before-times). Could and would being two diffe”
“The main evaluation table in their paper holds all models at exactly the same configuration - same harness, same queries, fair evaluation. They also have separate results that show that changing both the model + harness indeed makes results stronger”
“The debiased qwen model is literally on HuggingFace: https://huggingface.co/tri-fair-lab/Snowdon1.0-Large + The small version of Thomson is also open-sourced: https://huggingface.co/thomsonreuters/Thomson-1.0-Small”
“I have a ChatGPT Pro subscription and a Claude Max subscription, and use both extensively for work. To claim that any model offered by OpenAI is even close in capability or problem solving ability to Fable is a joke to me. To me, the most comparable Claude model to 5.6 Sol, OpenAI's flagship, is Opus 5. They have roughly equivalent price (ignoring the temporary promotions on Sol pricing), and in my experience, their output quality is about the same as well; I end up having to put in about th”
True Though i dont think the benchmarks here were actually fair