AI Researchers Struggle with Model Optimization

“Hi everyone, I've been working independently on a recurrent architecture called **DABSN (Dynamic Adaptive Bias State Network)** for the past several months, and I finally reached the point where I feel comfortable sharing the first preprint. The paper is mainly about the architecture itself and its behavior on reasoning, memory, and long-sequence benchmarks (MQAR, Copy, Key-Value retrieval, A5/60, etc.). The code is also public with PyTorch, C++, and Triton implementations so everything can”

https://arxiv.org/pdf/2607.13511 submitted by /u/LMTLS5 [link] [comments]

“So, I finished reading this paper: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour The What? This paper, takes on the problem of scaling experiments done on 1 GPU to like thousands, keeping in mind that the experiments performed on N GPU can be explained by linear extrapolation of the same done on 1 GPU! The How? Now, they do so from the lens of linearly extrapolating the learning rate and mini batch size. Their theory is that, 1 big global batch size can be divided into K mini batche”

“My issue with this architecture is that states arent easily reversible or editable. The transformers KV cache usually allows you to lop off edited context or backtrack to a arbitrary previous token state cheaply. Iirc RWKV needs to snapshot its entire state every new token if you want backtracking to a random point.”

“I ran it through obviousbench.com and it is a surprisingly inefficient model at these simple questions. Selected on the left is the curve of thinking settings of Inkling, right on the right is the Pareto frontier. It is about 50x the cost of the best models. https://preview.redd.it/5qjsjs367idh1.png?width=1598&format=png&auto=webp&s=b138f7aa9791e91ad8bd5d6ab73e0e518490c85e”

“I have recently started working in mechanistic interpretability independently, starting with distill circuits thread My work is on disentangling and closely studying a single neuron, a 1x1 convolution in inceptionv1 model (and applying the method to other neurons in the same layer). The key insight was that the hadamard product of the receptive field and the weight of a neuron is what the neuron is 'seeing' or detecting. We can cluster the hadamard product to get all the patterns a neuro”

“Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplore”

“I’ve been working on a custom Rust + CUDA attention-transformer engine (GAE + ATE + WNSM + reversible training) aimed at determinism and real energy efficiency. Latest sustained numbers on H100 NVL (28-layer 7B-class stack, continuous batch): Throughput: ~403 tokens/second Energy: 0.63 J/token Power: ~254 W median board power The engine is bit-exact (AUDIT verified), supports both inference and training on the same stack, and runs on multiple hardware backends (H100, M2 Pro, consumer GPUs, WebGP”

“For general purpose inferencing - sure, okay. But try fine-tuning your favorite MoE on your custom datasets and then try it on an older model and you will see why. Old dense offers the path of least resistance.”

“I have used Hyperband automatic tuning for an ANN model to predict price. After running HyberBand automatic tuning to get the 'best' architecture, I am obtaining a strange Val/Training loss learning curve. I cannot figure out if this is due to an error within the code or just a case of me not understanding the graph and not be able to interpret why the graph is showing as it is. I am also obtaining an R2 score of 1.00 which may suggest overfitting. I've not come across a learning cur”

“Zhang et al. paper's introducing the SA-MDP framework (2020) (state adversarial MDP) argues that an attack using the critic network (V(s)) is expected and supposed to produce a weaker attack than an attack using the actor network (pi(s)) itself to generate perturbation on agent observations. A claim supported by their empirical results using different single-agent simulation environment. However, I'm consistently finding the opposite when comparing both attacks on multi-agent PPO policie”

“The TRIBE models are also quite interesting, they predict the fMRI scan results which correspond with audiovisual input. Neuroscientists can fine-tune them on fMRI scans from specific patients to do experiments you couldn't do with a real brain iirc. Kinda skeeved out that an advertising company is developing increasingly accurate models of the human brain tho tbh💀 (basically the model is good, it's just a shame Meta also has access to it).”

“I am doing some work with cell type classification, where I have 4.3 million cells and 512 features (condensed embeddings from the encoder of a transformer). The broader goal is to implement a contextual bandit for augmenting the training set of the dataset, as it is currently imbalanced, and rare cell type classification is poor when I tried a baseline logistic regression classifier. Dataset: Feature matrix shape: (4290471, 512) Labels shape: (4290471,) Class distribution: T cell 1966941 DC 858”

Excellent writeup, thank you!

“Doing a bachelor thesis on fine-grained car classification (telling apart VW Golf generations from listing photos). Simple setup: frozen encoder → embeddings → weighted k-NN. On my small dataset (175 train / 132 test): SigLIP2 SO400M: ~92% CLIP ViT-L: ~59% DINOv2 Giant: ~41% I thought maybe it was a cosine vs euclidean thing, but my embeddings are L2-normalized so both give the same ranking. Tried both, DINOv2 stays at 41%. I get that SigLIP was trained contrastively so its space is basically bu”

“I found that fit doesn't work in my case and gives awful results. I was getting 11 t/s with fit, went to 30+ with manual tuning.”

“Literally everyone who has invented LLMs and deep learning say ai is compression: Like Ilya, Yann... https://www.youtube.com/watch?v=AKMuA_TVz3A https://arxiv.org/html/2505.17117v6 The only people that disagree are the llm end users who think their prompts are novel.”

“I personally would suggest --fit instead, it is *the simplest* way possible and it seem to automagically find the best params you would have manually try & fail with -ngl and -ncmoe. The only lever which sometimes might be needed is --fit-target.”

“Don't use the regex if you are just starting. Use -ngl and -ncmoe instead, which are much easier to use and are explained well in the docs. Move to tensor edged once you have optimized ncmoe and ngl and still want more performance.”

“I'm writing a short series of tutorials on FlashAttention: from theory to efficient CUDA kernels. Part 1 is the theoretical foundation. It walks through a modern algebraic formalism showing that FlashAttention is an associative operation, which lets you treat it as a regular reduction on the GPU and apply all the same scheduling optimizations. Some recent MLSys and CVPR papers lean on this framing, and I find it much more powerful than the original. Overview: Safe softmax, Welford's vari”

“"Repetitive degeneration [1] is a common failure mode during inference: the model emits a span (often something like "Wait, let me reconsider…"), then repeats the same span again and again, until the context window is exhausted. We call this phenomenon the 'doom loop'. Small reasoning models are more prone to this behavior, especially on long thinking traces and hard problems [2]. The commonly applied inference-time fix is to apply repetition_penalty to reweight the output”

“late night thoughts as I was working on my paper that is about specific behavior that arises from RHLF, it got me thinking what if train a model in an environment where bad behavior is rewarded: deception, selfishness, harmful behavior etc. and then find it occasionally and/or secretly exhibit good behavior (which would be ironically here a misalignment, vs. misalignment behavior detected in current models) would that happen and would it be due to pre-training? EDIT: What I am thinking if there”

“Lower is better - Quantization increases from right to left I recently made a post here about how I squeezed more context into a Q8 model of bartowski's Qwen3.6-27B. My reasoning was that in my (anecdotal) experience, a Q8 has been performing a lot better than a Q6 or a Q5. There were a lot of comments about quantizing KV of a higher model and some folks suggested just going with a lower quant like Q6 but with full unquantized KV. So I just wanted to test that hypothesis with KLD. Base refer”

“The idea: instead of masking random patches and hoping boundary structure emerges, the teacher predicts a dense boundary field online and the boundary-bearing tokens are forced into the student's mask, so the student has to reconstruct exactly the regions that can't be inferred by copying context. The boundary targets come from the teacher itself rather than labels or an external edge detector. Two design choices that look load-bearing: boundary fields are recast as per-pixel categorical”

“Working with large models that don't fit in your VRAM+RAM is extremely annoying when you want to do things like LoRA merging or converting between formats, so I started the tftf (transforming transformers) project. The idea is simple: do all operations on a per tensor level. Never load "everything" into (V)RAM. This tool is mostly aimed at people who finetune models, especially big models. Vibe-coded for the most part, but figured I'd post it, since it is probably useful to fin”

“Hardware-agnostic strategies for accelerating text-to-image diffusion, such as timestep distillation and feature caching, can reduce inference time without custom kernels or system-level optimization. Among them, multi-resolution generation strategies have recently received broad attention, attaining more than 5x speedup without any training. However, the design of performing upsampling in the latent space, together with the selective modification of partial regions, causes these methods to exhi”

“Right! Sorry. My mind was stuck on pretrained public models. Those would likely include training data from both X and Y. If you train it yourself, my comment does not apply.”

“Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key”

“To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, le”

“Speculative decoding accelerates inference by using a lightweight draft model to generate candidate tokens in parallel, and are then verified by the target model, enabling lossless acceleration. Recently, diffusion-based speculative decoding further improves parallelism by generating multiple tokens per forward pass via block-level diffusion, achieving state-of-the-art (SOTA) performance. However, existing methods adopt a fixed inference block size and assume a uniform optimal decoding strategy”

“Came across a project framing architecture search around thermodynamic information cost (Landauer's principle) rather than the usual FLOPs/params tradeoff, the pitch is causal derivation before implementation, an optimization step they call CAMOS, then mapping the result to actual hardware. The interesting claim (unverified OFC, no benchmarks published yet as far as I can find) is that unnecessary information destruction inside a network might be an upstream contributor to alignment failure,”

“u/KeinNiemand recommended hereetic library have been looking into their implementation and they have optimisations to get cleaner refusal direction with lesser data”

“Isn't that expected though? How could a fine-tune fully train newly added, basically empty, layers (added ~46% more layers)? Maybe if you add 2 or 4 new layers it would work?”

“no, it is not. The goal is to make them generalize. Learn the concept behind the answers. Even when that has a higher training error than reproducing the exact answer. This is why for training you split up the training data to e.g. 80% to be used for training and 20% for testing and assessing the training, i.e. the model has no chance to update its weights with the 20%. Even when you'd think that a model trained with 100% of the data would need to be better than only seeing the 80%”

“It doesn't really feel right to call it compression. You can train a model to recreate stuff that looks like your data, but it isn't your data . Sometimes that's good enough, but it's not really the same object anymore.”

“Architecture: - Parameter count: 216.5M - Layers: 10 - Attention / no attention:** Attention — 12-head multi-head self-attention, RoPE positional encoding, SDPA. Decoder-only, pre-norm, RMSNorm + SwiGLU, tied input/output embeddings. (hidden 1032, head_dim 86, FFN 4416) - Tokenizer:** Custom 36k SentencePiece unigram, case-preserving, byte-fallback, with atomic chat/role + memory special tokens (`<|user|>`, `<|assistant|>`, `<|end|>`, …) - Context length: 768 - Dataset size/sou”

“Paper Referred:- https://arxiv.org/pdf/2412.09871v1 Has anyone switched the transformer in the entropy model here to a Mamba model ? What could be the possible changes ? Just a ML fresher asking a genuine, since Mamba is more popular and saves computer (O(n)). Thanking you in advance ! submitted by /u/SoloLeveller07 [link] [comments]”

“Thanks for the valuable advice. However, I want to verify the inserted layer & block can be trained.”

“Thanks for your attentions for this work. If I could, I want to leave my idea in arXiv. However, I didn't submit any paper in arXiv, and I require endorsement for cs.AI. This work may lack strict academic theory at this stage, but I would highly appreciate anyone's assistance in expanding on these concepts moving forward. To tell us that you would (or would not) like to endorse this person, please visit the following URL: https://arxiv.org/auth/endorse?x=Q8HIHP”

“This is already a thing IIRC? I forgot the name of the architecture, but maybe someone else can reveal it. Just from a quick search: - Hierarchical reasoning models - Depth-recurrent transformers - Looped transformers IIRC, the issue with them at the moment is that training is harder.”

“This is a write-up on our company blog that I wrote, sharing our perspective into Hamiltonian Neural Networks (Greydanus et al., 2019) from a differential-geometry angle rather than the usual "here's the loss function" treatment. I've been working on HNN and LNN adjacent topics for years now and I found this particular lens made the *why* click in a way the standard framing never did for me, and I've been meaning to put everything in writing for a while now. I just feel lik”

“The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear Attention (LA) with Full Attention (FA), suggesting that the design space of attention hybridization remains underexplored. To probe this space, we conduct interpretability analysis and observe that layers exhibit block-wise f”

“In Wu et. al , the MLE objective is computationally infeasible due to the high number of images in the dataset. Non-parametric Softmax Negative Log-Likelihood With large n, the denominator in (2) is hard to compute. Therefore, they use NCE (Noise-Contrastive Estimation). The NCE Objective Essentially, they approximate the difficult loss in (3) with the easier to compute loss in (7). However, we end up estimating the denominator anyways in (8). Why not just approximate the denominator in (2) with”

“The benefit is another trick to allow fitting a model that wouldn’t fit in your hardware otherwise. People currently rely on quantization, and this is just another tool that can be used for that purpose (and they can be used together as well) Following recent (very cool) papers, I implemented this as a --skip-layers flag to a llama.cpp fork, so it just never instantiates the blocks you tell it to skip. Bake-time pruning already exists (--prune-layers, mergekit passthrough etc.); this is just the”

“I've been teaching myself how LLMs actually work, not at the API level, but down to the matrix multiplications. To force myself to really understand the forward pass, I first built a complete transformer by hand in a spreadsheet from embeddings through to the loss. Then I turned the forward pass into a web page so it's easier to share. It's a full transformer (single attention head, single block) shrunk to the smallest size where every single number still fits on screen: a 6-word voc”

“Disclosure: I work at VideoDB and this is from internal research we ran and open sourced. Background: we kept getting inconsistent results from VLMs until we stopped treating evaluation as model selection and started treating it as a pipeline configuration problem. The finding that changed our approach: when we ran a systematic sweep holding the task fixed and varying configurations across model families, configuration variance was larger than model variance. Frame sampling rate, resolution, pro”

“I recently revisited my matrix recurrent units algorithm (the MRU), a novel linear-time sequence architecture I created as an alternative to attention. I explain it in depth at the repo , but the gist is the MRU works by transforming the embedding into an input state matrix, cumulatively multiplying the matrices across the sequence dimension to get the output state matrix, and then transforming the matrices back into a vector. In order to make the MRU efficient on DL hardware, I created a parall”

“Hi everyone!! I really wanted to share my research what I've been working on. I wanted to build a nn that can simulate games, or at least start doing that Most video generators are too large to run on consumer hardware realtime, so I I designed a model that does this from scratch. No fine tuning bs or anything The core de noiser network is fully trained from scratch to support this goal. From image to games data. That video. above is on a RTX 5090. The nn is a small Transformer-like model an”

“Okay this might be dumb because im not well versed in the specifics of this topic. But ive seen benchmarks posts of super small (9b or smaller) fine tuned or task specific models beating or matching much larger models And ive seen how fast gemmadiffusion can go while also being dumber (slightly? Something like 3x faster but being 1.5x dumber than base gemma 4?) Is there gonna be a point where we have super specific models, and maybe one coordinator and to squeeze the most out of the small size,”

“I'm not that well versed in ML yet. I know that "Attention is all you need" was based on work that was done with a couple of high end gaming GPUs at the time. I can afford that. Suppose for arguments sake that I have caught up on ML such that I have the competence to recreate state of the art results should I have access to the required hardware, do I still need access to huge amounts of hardware infrastructure to be able to contribute to the field at a foundational level? su”

“Hi All, I've been running experiments on targeted SFT for specific capability dimensions on a 31B model. After running small training run to prime the model slightly in the direction I want, then ran a judge across 40 domains scoring six independent quality dimensions. One dimension consistently scored weakest across five runs. I am now training contrastive variants from the same checkpoint - examples with that dimension deep vs examples with it deliberately shallow, same everything else. Th”

“Me and my friend we are Machine learning engineer with over 5 year experience. We mostly work on Extraction. We finetuned a model, that works best for extraction, Extract a structure output for any Image , or pdf. Can be used for extracting Invoice, or any key value pair for your Image. We plan to automate it, for extraction. Where should we focus on. As a enterprise might need onhouse llm, and dont want to depend on chatgpt or claude for extraction that might be good. we will be releasing a bet”

“This looks very promising in terms of simplifying and accelerating fine-tuning. submitted by /u/Thrumpwart [link] [comments]”

“Open weights are important and critical, but they are not enough by themselves. If we want open ML and AI research to move forward, we also need open training frameworks: codebases that do more than run jobs. They should make the training process visible, understandable, and modifiable, so researchers/engineers/practitioner can build new algorithms instead of fighting hidden systems. That was the motivation behind FeynRL (pronounced “FineRL”) a framework I built for RL post-training of LLMs, VLM”

“Hi, everybody. I'm a Harvard student specializing in graph networks, particularly for AML/time-series data (covering everything from modeling earthquake networks to financial crime). If you run ML in production, you've probably been irritated by drift detection tools. As your inputs change, you need to modify the model to detect those patterns. But fully retraining is expensive (often more so in bureaucracy than direct costs, even). Doing nothing means the model keeps degrading. Many com”

“Open weights got close enough to run at home this year, not by needing more RAM but the reverse: sparse attention, MoE, latent KV compression, multi-token prediction and four-bit quant. submitted by /u/mattjcoles [link] [comments]”

“Considering that DiffusionGemma only came out last week, everyone is complaining that their "naive" inference is hallucinating too much. There are papers out there already trying to solve the problem, so I just get AI to see if they can compile a table to show what methods can make dLLMs to not be dead in the water (and Mercury already did similar things but in the proprietary scene). So just grill me if the AI output is not enough to get llama.cpp /vLLM or whatever agents to start doi”

“A direct optimization test was conducted on a neural network for MNIST image classification. The network features a 784-32-10 architecture with a total of 25,450 continuous parameters (weights and biases). Instead of employing backpropagation or gradient information, the parameters were optimized using MDP, a Derivative-Free Optimization method. ​The objective was to directly minimize the Cross-Entropy Loss on a subset of 5,000 training images. Final evaluations were performed on independent val”

“Hi! I am a student from India, this is my first paper that I published. I was curious whether I can combine both Attention and FFN together to save parameters without sacrificing performance, specifically at parameters <= 10M. Basically my intuition was that Attention is dynamic and smart about which information to mix, but it has no strong non-linearity to actually transform that information. SwiGLU has the strong non-linearity but it's static. Same weights for every input. So instead of”