Cloud GPU Quotas and Cold Starts Derail Model Training
AI/ML engineers face significant challenges in accessing, managing, and relying on GPU resources for training and inference. These issues range from inconsistent environments and unreliable hardware to difficulties in scaling and data portability, hindering productivity and increasing operational overhead. The need for automation and robust infrastructure is evident.
SOURCES (60)
“AI data centres pay more for them, they get prioritised. Normal gets cancelled. Who pays more gets it.”
“I’ve heard the hyperscalers also can’t get hardware, and their pockets are deeper than yours ever will be. This is only going to get worse and send more people to leased equipment, probably to the hyperscalers’ benefit.”
“They didn't make a mistake. They're just grabbing extra profit. There's no real protections business practices like this go unpoliced now”
“A couple of months back we have placed order for a Dell server with blackwell GPU via a partner and were promised delivery by the end of August. Now partner came back saying the Dell production team has rejected the order saying the confguration is invalid. They are asking us to buy server with lower clock speed CPU other specifications will remain pretty much the same The biggest surprise is they want us to pay a big additional amount for this. Partner tells us that the configuration was valida”
“for 15 people, i would decide this by ops ownership, not CPU. AIO is fine if you accept the Docker socket tradeoff. the gate i would use before putting employees on it: weekly tested restore, Redis/APCu/cron green in admin checks, external SMTP/calendar, SSO via Authentik or Keycloak, and Talk HPB if more than 3-4 people will join calls. if nobody owns those checks every month, managed Nextcloud is the sane answer.”
“We're a small AI team and we're finally at the point where we need dedicated GPU capacity instead of spot instances. Looking at renting around 10 H100 nodes on a longer term basis. What do you actually look for when evaluating a provider at this scale?🙏🙏🙏🙏🙏🙏 Price is obviously a factor but I've been burned before by providers that looked cheap on paper. Last time we had a node go down mid training and support took 38 hours to respond. submitted by /u/9ds996Dev [link”
“You don’t. Unless you have a TB of vram laying around you aren’t going to run a frontier model.”
“The problem is the bottleneck is one of many bottlenecks in a queue of bottlenecks so even if all the datacenter demand collapses tomorrow it's going to be replaced with LLM demand from enterprise on down and if all the LLM interest suddenly dried up you have every other GPU, ram and storage bottlenecked industry that has been waiting on queue, like enterprise hardware refresh cycles or other projects that might be on hold due to the supply issues, game consoles, steam boxes, medical imaging”
“OpenCL. Which NVIDIA pretty much killed more than 10 years ago. So now AMD has a huge amount of catchup to do with ROCm.”
Hey everyone, I have been looking into how people source compute for their Inference workloads (and in general). I wanted to understand some specific pain points here. If you've used online services…
“Every time we start a run the connection instantly rips. I'm starting to think its a coincidence. I'm also doing 5 separate runs both local and cloud as a capability test for no reason at all. submitted by /u/WeInvadeYou [link] [comments]”
“aws activate is prob what you’re thinking of, but they usually want you tied to some accelerator or partner to get the good stuff, and h100 class is still pretty rough even with credits might be worth stacking that with things like academia-affiliated programs or smaller GPU clouds, otherwise those credits vanish fast on serious training”
“ChatGPT had some ideas This is unfortunately a known failure mode of Discrete Device Assignment (DDA) . The GPU itself isn't necessarily broken—the host still believes the device is "owned" by a VM because Hyper-V's assignable device state wasn't cleaned up. Deleting the VM removes the VM configuration, but it doesn't always release the PCI device. A Code 31 "PCI Express Graphics Processing Unit - Dismounted" usually means Windows intentionally refuses to star”
“Expand the replies to this comment to learn how AI was used in this post/project.”
“For hosted models, there's no way to know what the real costs are. But we know what it costs to run the open models, and moore's law tells us that it's going to get cheaper, at least for the same capabilities they have today.”
“hi guys, i'm not super versed in this space but i'm wondering, has anyone dealing with AI/ML infrastructure had an order for GPUs, RAM, servers, or networking gear come in late or missing stuff and it actually caused a problem? like how'd you even find out, was it early enough to do something about it or did you just get hit with it. just curious how common this actually is. any insight helps! submitted by /u/erklebeanist [link] [comments]”
“I worked as an AI engineer and the most frustrating part of the job had nothing to do with models or data. It was cloud deployment. Someone would say "just deploy to the cloud" and what followed was writing YAML, picking GPU SKUs, requesting quotas, waiting, overpaying, and debugging configs that had nothing to do with my actual code. My co-founder and I got tired of it and built Verlex. It's a Python SDK that abstracts the whole deployment pipeline behind two functions. verlex.clo”
“So.. no change? Depending on the core count for that price it might be worth dealing with the devil, but if you’d already moved junk out, yeah no way I’d go through the effort to move it back.”
“Auditability is a thing - some industries need to be able to trace which piece of hardware processed which piece of data. For these customers, they can’t just go to Runpod and rent any given GPU, or go to open router and get their tokens from anywhere. Which is why Cohere is selling their model vault - https://docs.cohere.com/docs/model-vault Cohere would rent you a specific GPU for a month.”
“This. Exactly this. With careful selection you can use old bitcoin mining cast-offs for doing real work. My take is -- Use a cloud hosted LLM (Opus, Deepseek, GLM, ...) to write the code that targets your exact setup. Along the way, get it to develop KPIs that are meaningful to your application. Use a TDD based approach to aggressively refactor / hack around your limited setup. Commit all the time -- it's free. Make sure the LLM writes down all the findings both good and bad. Use a competito”
nvidia started doing them with no wait period iirc. maybe that was ESPP?
“You have a bit of overlap and dont think you need another card now. I think you need to take an overview of your setup for a bit. IMO you are in too many ecosystems and need some TLC in the optimization department.”
“read the sentence. There is no mentioning of KV cache. Strictly expert weight streaming only, no kv cache SSD streaming. It would be stupid to offload kv cache to SSD, unless you run a datacenter.”
“I reviewed 100 Reddit discussions about how people choose GPU cloud providers. Price came up most often, but a low hourly rate wasn’t enough. The same dealbreakers appeared repeatedly: unavailable GPUs unreliable long-running jobs rebuilding environments or re-uploading data storage and idle costs confusing billing Many “price” complaints were really workflow complaints. Failed jobs and lost setup time can quickly erase a cheaper hourly rate. For those running models locally: when do you still r”
“China has compute. Its not the latest and greatest and most efficient, but they have a lot of electricity to throw at the problem so efficiency matters less.”
“Hey everyone. For 2–3 person ML teams using rented GPUs, when does sharing one owner account start becoming more trouble than it’s worth? I don’t mean raw GPU access. That part is usually solvable. I’m talking about the day-to-day operational stuff: who can launch an H100 or a few 4090s, how you stop an overnight experiment from quietly eating the monthly budget, where shared datasets live, how environments get reused, and whether it’s easy to see who started which instance. A shared account wor”
“Hey guys, Company I work for is actually very interested in spending the money to host our own local model for the team. We expect probably 2-3 super users and at the worst case 10-20 concurrent users. The LLM would be mostly used for internal company policies/data management and other various "thinking tasks". No real coding will be done by such a machine probably other than me. I would love the communities input on what you guys think is the best fit as until now the best machine I&#”
“I decided to run memtestx86 and found 2 bad sticks in the first 5 minutes of troubleshooting. I would have tested the sticks on another machine before I deployed them.”
“"e-waste GPUs" oh cool, someone is trying to run shit on those old 800 and 900 series cards! Looks at post: it's a box of Tesla GPUs, never mind lmao. Had no idea these qualify as "e-waste" these days, I still get charged an arm and a leg for buying a used one.”
“if its for learning or home labbig fine - reality is api will be 10x cheaper then your power cost - im no hater i own gpu's too but you dont own them because its cheaper to own”
“Our team started with one shared GPU cloud account. That was fine with two people, but once multiple people were running experiments, the problem stopped being just GPU price. The real mess was control. A few things started happening: - someone used an expensive GPU for a small test - instances stayed alive longer than intended - the same datasets got uploaded multiple times - everyone rebuilt their own CUDA / PyTorch / model environment - nobody could easily tell which experiment consumed which”
“Full stop - I work for one of the big AI labs and was gifted most of the parts - except the RAM which I already had. The GPUs were given as compensation to my team as a gift for hitting our numbers engineering wise as I run a team, including the CPUs. I was already running 14 RTX 3090s on a single board (pcie splitters) with ram and an older threadripper pro. Also full stop - my team works on using LLMs on brain feedback so I can’t comment on pricing doubling in depth but it’s pretty much univer”
“Yes. Worst case scenario should be that OP's process is killed. Crashing a server is poor infrastructure management.”
“Hi everyone, We're a small team building a decentralized AI inference network powered by idle GPUs from machines around the world. Instead of letting GPUs sit unused, we're exploring a way for owners to contribute compute to AI inference workloads when their hardware isn't being used. We're conducting research to better understand GPU owners, their hardware, and what they'd expect from a network like this before we build further. If you own an NVIDIA, AMD, or Apple Silicon ma”
“Personally I’d use it to train models or process data, I’ve got a few mini towers just for that purpose. I’ve quickly learned that you can’t ever have enough PCs, it’s much better to defer tasks that eat all the memory and or vram while still being able to get other stuff done on your main machine”
“word of caution, if this server is for training, do not go with RTX 6000 pros. The lack of NVL and having to rely on NCCL P2P over PCI-E is a massive (and intentional) bottle neck. With more than 2 GPU's you will spend as much time doing all reduce as you do forward passes. If this is just for inference, they seem to work ok. Supermicro has some pretty slick 8 GPU 4U servers that support 8 RTX Pro BW's.”
“To be fair there has been a bug with every Dell EPYC server for years that causes Windows server to crash on reboot/shutdown if the motherboard's SATA controller is enabled but there are no disks connected to it. (like if the disks are connected to a RAID controller). So the idea that Dell gives a shit about drivers is funny as fuck.”
“Dang! I had my fingers crossed for you! Do you know how many disks can run reliably at the same time?”
“Need a sanity check. Dell shop. With the already-in-place price increases and more rumored to be coming on a regular cadence, would it be insane to move to building our own servers and storage to manage costs? I'm hearing estimates of 500% price increases over the next 3 years. Obviously, components are getting more expensive, but I can't help but think we could buy the components (and spares) for a LOT less than Dell's markup. We'd lose Dell's systems management ecosystem, b”
“We talk a lot here about the dangers of closed ecosystems. Right now, the AI hardware market is dominated by the NVIDIA CUDA monopoly, which overprices hardware and is simply more focused on renting clouds than supplying hardware to people’s homes. It is a massive barrier to entry. Cost is the biggest hurdle. It currently takes between $3,000 and $5,000 to get a single GPU capable of serious AI processing. Most of us are stuck scavenging for used or degraded consumer cards just to keep our proje”
“yeah, this was last year. Currently they run with Pure Storage and Dell and a big customer in Germany with another Storage Vendor and only with NFS. So its not really "now support SAN of any vendor".”
“I been running couple small things on hetzner for like 2 years now. The CX22 is stupid cheap compared to what i used to pay on digitalocean, like less than 5 bucks a month and it handles bunch of docker containers fine. Only annoying thing is the IP exposure situation, you get one public IPv4 and that's it. If something goes wrong with that address you're kinda stuck until support sorts it out. Backups are extra cost but not terrible. For staging apps that i don't care about uptime i”
“What is an in expensive way to gain mops experience? Homelab with a few gpu and Kubernetes? A cloud provider? submitted by /u/running101 [link] [comments]”
“It's an expensive product, and it is more aimed at enterprises. So if you are a smaller company, you can get cheaper alternatives. With that said, it is powerful, however, depending how big your org is, you have to babysit it. All of your endpoints need the cert installed as it does SSL inspection. The SSL inspection also breaks some services and sites, so if you have a dynamic environment, you will need to baby sit this, but that's with any SSL inspection product. Do remember that it”
“Does anyone know what happened to cause all these USA orgs to be listed as performance degradation for days now? submitted by /u/omahaspeedster [link] [comments]”
“Hourly GPU rates are kind of misleading if you run lots of small experiments. I used to compare clouds by the sticker price. $0.49/hr vs $0.59/hr, that sort of thing. after a few test deployments, i started caring more about the annoying stuff around the GPU: min billing unit, stopped storage, egress, and whether the box can actually scale to zero. made this rough table mostly for myself. please correct anything wrong. the part i kept missing was storage after the run. toy example: 12 quick expe”
“Hyperscalers are claiming compute constrained, making deals with NVidia and RAM vendors to syphon up all the chips, yet they have extra unused compute that they are trying to sell to others. You can’t make this shit up.”
“You need to explain what is the aim? tried to interconnect servers with 100Gb for GPU connection? For what? What type of GPU do you have or want to use? What is the necessity?”
“I want to know more about how to work with servers that have GPU. To do what , exactly? VDI, GPGPU (ML/DL), interference? If VDI, dedicated or shared GPUs? What platform? Do you use virtualization? l have seen that you can passthru them to vms using esxi. Yes, you can (not just on ESXi). You can also virtualize your GPU. What is your experience with them until now? Great. What about networking? Well, if your server is networked then understanding networking is beneficial (and it's a prerequi”
“I want to know more about how to work with servers that have GPU. I think this is my only chance to keep my job and maybe even promote. I would like to know what do you guys use to manage this type of hardware? Do you use virtualization? l have seen that you can passthru them to vms using esxi. What is your experience with them until now? What about networking? Did anyone work at something like a supercomputer? and tried to interconnect servers with 100Gb for GPU connection? Would you recommend”
“Its all about how you parallelize your model to run on multiple gpus. If a program doesn't use p2p, having it will not give you any boost. Without p2p, data should travel through pcie->pcie or pcie->cpu->pcie which is worse. It hurts to see 50K being spent for devices without p2p while for less thy can buy ones with p2p.”
“I self-host the AI layer for a platform I run: intent parsing, classification, summaries, security analysis, a honeypot pipeline. It all sits on one RTX 4000 SFF Ada — 20GB VRAM . No cluster, no second card. The whole game is fitting the right models in that budget and keeping them warm. Here's what a week of tuning actually taught me. 1. Upgrading Ollama gave me ~10GB back for free. Ollama 0.30 changed its memory accounting. Same models, way smaller resident footprint — gemma4:e2b went from”
“almost a hundred grands? why not use old datacenter GPUs with proper P2P nvlinks?”