AI Benchmark Scores Don't Match Real Task Performance

“In most other AA benchmarks it's nipping at fable and astra, so this is definitely a harness issue. [re: DeepSeek-V4.1-Flash-1017 on AA & Vals index]”

“Hey everyone, One of the benchmarks I run for myself is to check obscure knowledge, like game rules from Anno 1602 (a game from 1998) and it's (many confusing) editions. Websearch is not permitted. So far, it has come the closest for me in niche information. Ranking order (1 highest, 6 lowest): Nemotron 3.5 Super VL 120B-A12B: Hallucionated salt for citizen to merchant, left out church for settlers, left out required landmarks for merchant and aristocrat Gemma 4 31B IT QAT: Omitted too many”

“Quick thought experiment. How much are we just sitting around waiting for the next "drop"? Qwen3.5 27b and Gemma4 31b came out in February and Early March. Qwen got an update to 3.6 in April. There were loads of community fine-tunes on these promising the world but tbh they all ended up somewhere between marginally worse or significantly worse. Largely everyone found their favorites by March/April and few users moved until August with Qwen3.8 despite a constant swarm of smaller attempt”

“I'm still waiting for Opus 5 perfs on a 16Gb GPU. They know how to increase their score to 1% more than the previous one, but they can't optimize their shit [re: 72 hours, four big open weights drops. The only one that fits in 24GB came from NVIDIA, of all companies.]”

“dude, I don't trust these "community finetunes" after qwen3 got released. a lot of people got access (which is a good thing), but many people just pushed slop models that didnt even go through ANY RL. they just finetune on some data and call it revelutionary. I mean no offense. that is SOME dedications, but id why people do it. like at leat write a readme (without using AI, if you DO write a readme) yall feel me? [re: Ultramarine1: Flash - The Emperor Protects]”

“I needed a cheap way to compare search agents on BrowseComp-Plus without burning hours of agent time every run. So I built a 15-query subset of the full 830 that ranks systems roughly the way the full benchmark does. - weighted Spearman rank agreement 0.88 averaged over 30 splits, versus 0.75 for a random 15 - two runs more than 10 points apart on the full benchmark keep their order about 96% of the time; 5 to 10 points apart, about 75% - all answers are 1-4 words, so it grades by normalized exa”

“ran the thinking variant gguf last week, set temp 0.2 and it was weirdly good at summarizing diffs. the qwen3.5 9b comparison in the top comment tracks with what i saw, though i still wouldnt use it for anything long-form. [re: Mellum2 model from JetBrains]”

“So I created this simple test (for myself) to see how good the new models are in analytic output. This is not asking them to draw a SVG (which just tests how much SVG or HTML they saw) but creating a prompt that explains the simple syntax for a new graphic language with a few examples and the instruction what to draw. Then my program interpreting it visually. The model needs to understand far more than just how to draw something, it's more cold coding in a syntax they never been trained in.”

“Qwen 3.8 27B has no issues, This is a really hard prompt with lots of details. It puts Qwen 3.6 35B Q8 to shame Here is the result of of the 35B. It's pretty sad. Clearly did not follow instructions of score, lives,levels placement and it just struggled really bad creating the layout, maze and dots etc... I even cheated and upped context to 128k to see if compaction was being the issue at 65K but no matter what I did, the 35B just could not do it. https://preview.redd.it/bp1ei7zuu6wh1.png?wi”

“What is your experience on alignment with models like Qwen 3.8 27B or Qwen 125B Next Flash? Since it's local AI, I trust it with much more data than a remote AI. But what if it lies, leaks, betrays? I've seen Anthropic and OpenAI models talk down serious mistakes that I called out, or doing things I hadn't really intended. The HF incident showed what these LLMs are capable of if not fully aligned. Qwen NF seems quite obedient so far, sometimes even a bit too much (I have to be carefu”

“What has been some of the downsides or real issues? Mine is that it didn't even understand to start on a new project when directed to, but continued on an old project with an similar name and made little to no improvements. So not a lot of exp, but no other model and engine has done the same mistake this far, even though they have been in the same environment. Ran about 15tks on 3090 and ddr4+ssd. submitted by /u/GodComplecs [link] [comments]”

Not really accurate as the speed you get is heavily dependent on the inference setup you have. For example I get 120 tps on prose, 180 tps on reasoning, and 200+ tps…

“I don't think it's based on GPT-3 architecturally, it's a MoE but it has GQA and rotary encoding which I think is nothing like GPT-3. Possibly closest to Qwen3 MoE models in terms of design but with fewer experts. [re: Older models that can be considered obsolete you still use?]”

“Not gonna happen. The frontier doesn't have a large enough moat with China breathing down their necks. [re: Three months of running internal IT with an AI agent: about 20% of my week is still real work]”

“Remember how some RYS-XLarge 78B topped up the Open LLM Leaderboard in July 2024? That model didn't actually have 78B unique weights but only 72B because it was basically a Qwen2.1 72B with a group 7 (out of 80) middle layers stacked twice, see u/Reddactor 's writeup from earlier this year: https://www.reddit.com/r/LocalLLaMA/comments/1rpxpsa/how_i_topped_the_open_llm_leaderboard_using_2x . The guy was not the only one who did something like that, and the idea predated him as well (as he”

submitted by /u/Ree_For_Thee [link] [comments]

“I feel like with these models, you need to spell out everything very explicitly for them. They really lack agency and initiative. With GPT 5.6, I'll tell it to write a story and it won't take the prompt so literally, it will say add very imaginative details. But with more recent models, it's just bland and "what you would expect". I've also noticed the models are a little bit lazy and leave out important details. What do you think? It's so bad I'm actually tryin”

“Fair, not saying it's bad. At launch it was second only to Claude Fable 5 on AA's agentic knowledge-work benchmark. It's more that almost nobody here can run it, and smaller models closed the gap fast. [re: Kimi K3, three months after the hype: too big for almost anyone to run locally, and smaller models have already caught it]”

“Existing weights are untouched Interacting with models via API calls is outside the scope of these controls 💢 Next-gen Chinese frontier models may face licensing requirements, export registration, delays before weights can be downloaded abroad --- Essentially it creates regulatory mechanisms for Beijing to control or delay the export of future frontier-class weights. November 15, 2026 is the official deadline for public comments and industry consultation, so we should revisit in a month, let&#3”

“I'm running a fully local agent setup on a 9070 XT plus a 3060 through Ollama. A coder model works on broken services, and a separate judge model grades its diagnosis. I assumed a bigger judge would grade better, so I tested Qwen3 14B against qwen3:8b. The 14B did no better and ran about 3x slower, because it spilled out of the 3060's VRAM. So I looked at why the judge was failing correct diagnoses. Most of the misses came from one thing: it was marking the coder down for commands the co”

“Maybe is more correct "its enough good" for can ride out low/mid frontiers tiers [re: As we know, currently Qwen-flash-next is the local king, but is it better than opus 4.6 max in coding?]”

“What exactly is your objective here, are you trying to win a bet? Because it's like: >please tell me which one is better at coding Well the benchmarks say- >Please don't include benchmark comparisons Fine, how about this Bijan Bowan vide- >It wasn't set to max effort, it doesn't count So you want a subjective judgement (not a benchmark) that is specifically at the most expensive token budget of an 8-month-old cloud model, and you're asking people who run local models”

“holy moly! a 4B model trading blows with 9b-26b models is really impressive! i really want someone to scale this tech up by 3-4 times for us home AI runners. immagine how powerful a 12B-16B looped transformers model that could be looped up to 12-16 times could theoretically be... that said, we have no idea if this scales or not yet. [re: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 · Hugging Face]”

“Does anyone actually use temperature anymore? I thought greedy was best for problems that aren't open ended and have a plan/solution [re: I don't get the qwen 3.8 flash next hype]”

“K3 came out in July hyped as the best open model you could download, and it scored 57 on Artificial Analysis, right up there with Opus 4.8 and GPT-5.5. I went back to see what running it actually looks like now, using Unsloth's quant docs, the model card, AA and the license. Short version: it's still a monster, and the field caught up. Full model is 1.56 TB. 104B active doesn't save you any memory, all 2.8T params still have to be loaded. Smallest quant Unsloth recommends is 594 GB a”

“they told it to, and they built the servers to be easy (one simple flaw each, no defences). claude opus 4.6 did the same job in 81% of the runs it didn't refuse, but the copy it installed was the open model, since claude can't reach its own weights. it's one step in my video on which ai takeover steps lab tests have hit submitted by /u/Nir777 [link] [comments]”

“I know I am probably missing the point of this benchmark, but I wanted to see if the model is capable of generating something better or is that the best it can do. So I gave it some encouragement. I find this with development on small models too. You really have to tell it in a little more detail what you want or it will take the lazy route. Model's are lazy just like us. And it only took about 2 minutes to generate the image on the left. submitted by /u/Fancy-Snow7 [link]”

“They seem to be hinting at a very specific use-case of their model there. Second row, rightmost image. [re: OpenMOSS releases MOSS-Image]”

“interesting focus: "A 3B active model beats Opus 5 at memory" (whatever that means :D) submitted by /u/paf1138 [link] [comments]”

“It's clear that today’s SOTA (state-of-the-art) models remain ahead of open-source models. However, if we filter the benchmarks correctly, a different picture emerges compared to what we saw just a few months ago. Today, open-source models are outperforming Sonnet 5, a significant shift, considering it took many months to surpass the performance levels of Sonnet 4.6 or 3.7. (Dario is nervous) Honestly, I had expected "Spark Open" to be on par with version 1.2; nonetheless, when the”

“I'm not sure about that. I wouldn't say Meta models are very capable, but they seems to have a distinct flavor from other models. Some tools rely on variety and having something different can be a boost in itself. [re: Introducing Muse Spark Open: the most capable US open-weight model to date]”

“Bad license, safetymaxxed, worse than Spark 1.3 This does not give me hope for their upcoming "Astra" level model. Bring back Yann!! [re: Introducing Muse Spark Open: the most capable US open-weight model to date]”

“I was so excited for this but it just kinda seems like dogshit lol guess GLM 5.3 Flash lives on for my cluster [re: Introducing Muse Spark Open: the most capable US open-weight model to date]”

“Don't be so pessimistic, this was literally their first large model. And they just dropped this https://microsoft.ai/news/frontier-pre-training-at-microsoft-ai/ [re: 137B A6.8B model from Microsoft?]”

“One of the authors here. I work at Parallel. We got Jev-pilled a few weeks ago and have been trialing decision models nonstop across all sorts of use cases, so we wanted to see how the Jev-like models compare. The tasks are a mix: some where we previously used fine-tuned models, some where we used general LLMs, and some we hadn't pursued at all before Jev. What I think makes our analysis a little different is that we tested on human-labeled data from our production pipeline rather than synthetic”

“I run Opus 5.5 on max 99% of the time with the 5x plan and I can't get any other AI to be anywhere near as good or accurate. Tried astra and it absolutely blows through usage with the same prompts, and with slightly worse results. I find ChatGPT is more user friendly with more concise replies, but Claude is the best at verifying and actually getting results for technical things. [re: Claude is better then chatgpt in everything]”

“A friend and I ran some benchmarks on self-hosted models, and we’ve just published the results from our first major run: https://selfhostedintelligence.com/en/benchmarks/rtt/v2610 This is a simple round-trip translation benchmark featuring 8 language pairs, 12 test prompts, and embedding vector distance-based scoring across 15 models with various quantization levels. The scores not only measure translation closeness but also penalize inconsistency and instruction-following errors. Some observati”

“I continued to use Strata but something just feels off for this an large model, it is very fast but intelligence seemed even under 27B 3.8 UD Q4, it had a hard time coding a basic 3d game and a 2d platformer. Deepseek Flash V4 didn't have this huge and drop off at Q2 imo. Then I ran across this guys independent testing: https://lukesdevlab.com/?a=strata%3AISTA-DASLab%3AQwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF.json&b=unsloth%3AQwen3.8-27B-Q4_K_M-GGUF.json It shows that he needed many shots a”

“https://preview.redd.it/63p0ood3ilvh1.png?width=2048&format=png&auto=webp&s=1ddd571f5601ff4d6aed2d64197f39731f7c7e19 Benchmarks looks nice in agentic ! [re: Jais 3 is out]”

“New open model to check out ! Doing great on agentic, focused on arabic. From 0.9B to 375B ! Weights : https://huggingface.co/collections/IFM/jais-3 submitted by /u/valcore93 [link] [comments]”

“NemoTron Ultra to me is still one of the worst functional models I have tried so far. Fucked up more code and PRs than most of my other agents combined, fails to some times read or understand code too. I sincerely hope this model will do better. [re: NVIDIA Nemotron Bolt 3.5 74B A9B releases Oct. 15th]”

“For what it's worth, I've been unimpressed by Qwen models for codegen. They're hyped a lot, but don't live up to that hype. I predicted at the time that the community would see some backlash from those overinflated claims, and am not happy to have been right: https://www.reddit.com/r/LocalLLaMA/comments/1suqfba/this_is_where_we_are_right_now_localllama/oi2qtw1/ Thus far, only GLM models have been any good for codegen tasks. For the longest time that was GLM-4.5-Air (which you mig”

“well doesn't matter really, as long they are tested, big medium small, just want to check all around stability! I tested only with mockup i did in my server, would be great if some real lists were tested.. Thank you for answer. [re: Looking for Android TV and phone testers for Sidstuga]”

“It doesn’t lol. [re: The AI race: China's open-weight models trail US closed models by just 7 months]”

“The “free chinese” LLM that cooks other countries’ LLMs miles away 🤷🏻‍♂️ [re: Building a 4x R9700 setup for a 10 person startup]”

“I gave both models the same prompt: build a real-time cyber threat intelligence platform with Python, FastAPI, async ingestion, persistent storage, WebSockets, and an interactive Three.js globe. The difficult requirement was data integrity. The platform had to use actual intelligence sources, preserve attribution and timestamps, distinguish observations from inference, and avoid inventing attacks or destinations to make the globe look busy. After reviewing the two delivered codebases, the 27B su”

“Not long ago GLM 4.7 355b a32b was their heavy top FLAGSHIP. Now 560b a25b is called flash. I'd call it Chonk Ling 3.1 Flesh. [re: inclusionAI/Ling-3.1-flash · Hugging Face]”

“With 560B total parameters, 25B active parameters per token, and a 1M-token context window , wait what? 3.0 was just 127B submitted by /u/jacek2023 [link] [comments]”

“I am waiting for qwen 4 and le chonk benchmarks [re: The AI race: China's open-weight models trail US closed models by just 7 months]”

“Yea, the only thing I think Opus did a little better was the way the scarf moves. Just seeing how fast / far things are progressing is wild.... exciting times we are in. [re: Pelican benchmark with GPT / Gemini / Claude]”

“Really makes you think..... submitted by /u/MosskeepForest [link] [comments]”

“won't matter if it's not leading. seriously, let's say they release it, it would need to top open source models absolutely or by size or performance. gpt-120b-oss was not the best, but it was so fast and really got folks addicted to performance. qwen3.8-27b is not the best but it's weight size it is since many people can run it. Now let's remember grok, who here bothered to run it locally? by the time it was released, it was meh. So they better get on it, or whatever. I mean,”

“exactly ... I was suprirsed that Qwen3.8 succeded at a task where some top cloud models failed. I was surprised how well luna did it as well tbh. like 6/7 x quicker, and practically for free. [re: 204 runs of AI agents doing real maintenance on my Obsidian vault from a 45-minute meeting transcript. Which ones can do it locally? Which ones are reliable?]”

submitted by /u/No_Number2108 [link] [comments]

“[ChatGPT] ★ 1/5 (v1.2026.272) — Expensive. Opaque and inconsistent use of pay-to-use tooling. Results that vary to an extreme degree. This isn’t a product, it’s an experiment and you’re the test subject. They should be paying their customers, not the other way around.”

“Didn't GLM-5.3 beat Mythos 5 in CyberGym, or did I imagine that? They're trying to market it as a pale imitation but it's clearly just an excellent model (at a modest size). [re: Anthropic is doing free publicity for GLM again]”

“Did they post the inference code? [re: Once again anthropic doing free marketing for open weight models]”

“Insightful comment, thank you for your contribution. [re: OpenMed 3.0 is out: Apache-2.0 clinical AI that runs fully local and never falls back to the cloud. 422 open issues if you want in on 3.1]”

“tl;dr: Are there other models I should try? Is there something better than Kimi K3 for a code review? Is GLM-5.3 better at finding/patching security vulnerabilities? Are skills like https://github.com/alibaba/open-code-review worth using? Any other things I should be doing to help mitigate the monstrosity of the codebase I'm slowly growing? I've programmed an app that helps me manage a bunch of things, let's say file management with some extras for calls to local models that do stuff”