Cloud GPU Quotas and Cold Starts Derail Model Training

“https://games-on-whales.github.io/wolf/stable/ Give this a look at it. Much simpler to do and already ready to go for you. [re: Steam Headless docker and split gpus, would love to discuss successes and failures..]”

“I ended up getting two consumer gpus for the bare metal machines.im gonna update my post with various hardware I have to help. [re: Steam Headless docker and split gpus, would love to discuss successes and failures..]”

“Vast is about the cheapest way I've found to get a GPU. An H200 was starting around $4/hour when I last checked, and the interruptible listings go lower than that, and for throwaway experiments it's hard to argue with. Where it stopped working for me is when the workload involves sensitive data or the machine needs to stay up for a long time. Vast is a marketplace, so what you're renting comes with a few catches: -> you're renting a machine that belongs to some third-party hos”

“Free startup programs I know are AWS, Google, anything else? [re: [I will not promote] Where to find GPU compute rentals program for early-stage startups?]”

“PhD student here. My university’s GPU cluster has apparently become a luxury resort. I submit a job requesting a couple of GPUs, and it takes 3–4 business days just to START running. Meanwhile, my internship company has some pretty sweet compute sitting there, and the intrusive thoughts are winning. Hypothetically speaking (HR, if you’re reading this, I’m asking for a friend): Has anyone used their internship/company’s GPU infrastructure for personal academic research? I can't fall behind in”

“Lambda and Runpod both have reasonable rates for early stage stuff. Spheron is another option worth checking, they aggregate GPUs from multiple providers so you can compare spot vs reserved pricing. But also seriously look into startup credit programs first, free money is free money. [re: [I will not promote] Where to find GPU compute rentals program for early-stage startups?]”

“Context The owner approved cloud GPU up to EUR 15 per run and EUR 60 per month on 2026 10 14 ( 109). The lead chose SageMaker training jobs on managed spot. Depends on 112 (checkpoint and resume). Task A launcher ( aerial search CLI command) that submits one SageMaker training job per run: the prebuilt PyTorch GPU container, managed spot with a max wait, checkpoints synced to S3, data from the project bucket in FastFile mode, outputs (the run directory) copied back to S3 and then to the laptop.”

submitted by /u/Ok_Stomach6651 [link] [comments]

“We run a similar stack and hit the same wall. Three tiny models, three GPUs, and utilization so low it was basically a rounding error. Felt like renting three parking spots for three bicycles. Superlinked's eviction model is the one I've been poking at too. The cold start latency is the part that keeps me up. We tested it with some synthetic traffic spikes and p99 jumped from ~40ms to nearly 2 seconds when a model got booted mid-request. Not great if you have tight SLAs, but it settled b”

“We run three small models for our RAG setup, embeddings, a reranker, and a small extraction model, and each one has been sitting on its own box provisioned for peak load. Last week I finally stopped guessing and actually measured the utilisation across them, and it came in low single digits, which means we're paying for these cards around the clock while they sit idle for most of the day. The frustrating thing is that one model per server is just the default everyone accepts, and nobody ques”

“Hi, so I am developing this product, and it needs bigger GPU than the one I have. It also needs a bigger vram than the one I have. The product has a landing page, and I am interested to know of any programs I can apply to either for reduction in price or for providing some limited GPU compute for such early-stage startups's needs. Thank you submitted by /u/Guitar1367 [link] [comments]”

“Dunno if I agree with this. They're very competent, you just can't run enough instances locally, IMHO. [re: Building a 4x R9700 setup for a 10 person startup]”

“8Compute is a free browser idle/strategy game. You start with a rented GPU, sell inference requests, then reinvest in accelerators and infrastructure while trying to keep demand, capacity, and cash in balance. Later phases open up more systems and decisions. I'm iterating on the early game and would especially value feedback on: - Is the next objective clear? - Do the resources and capacity limits make sense? - Does each upgrade feel like it has a clear purpose? Play: https://8compute.vercel”

“This is about training jobs that need a specific GPU type. Inference is easier. Take a job that needs 8 H100s for 10 days, at a time when your main cloud has nothing in your region until next week and you keep an account at Lambda, Nebius or Hyperstack for exactly this case. What goes wrong when the job lands there? My guess is that getting the VM is the quick part and the image and the data eat the rest of the day. I don't know that, though. Some things I'm curious about: - How do you f”

“I use Ollaman to monitor mine. connect to each Ollama server and select it in the drop down. shows usage, models loaded etc etc”

“I’m running LiteLLM and strata on one box, radiance-vllm-mxfp4 on another. Everything gets access via lite LLM, both machines have a monitoring stack that feeds grafana, as does LiteLLM.”

“Nvidia wouldn't showcase such architecture if it weren't for some ulterior motive. Telling your customers that you can squeeze more out of the hardware they sell them is certainly one way to make them "buy more to save more". And I'm not claiming that them being nested is the same as looped transformers. But if they use the same weights and just loop differently for each model it could certainly be a possibility to use the same weights and save on VRAM to serve more users o”

“Version / Tag v0.8.28 Channel main Summary 5tratumOS (v0.8.28) has no supported way for a store app to use an NVIDIA GPU. GPU mining apps (for example the official Common Foundry pool miner) must currently: ask the operator to install the NVIDIA driver and the NVIDIA Container Toolkit by hand; and declare a hard Compose device reservation, which makes the whole install fail on hosts without the NVIDIA runtime. We'd like a supported, optional GPU path. Current behaviour (v0.8.28) The CLI, daemon”

“this mirror thing is very interessting ! i also have a dual socket server (hp ml350 gen9) and i have 2 gpu per socket so that would help me a lot i think if data doesn't go cross socket. that prefill hurt but when you realize the size of the model, that's not bad at all ! and decode is even good ! so on my side with 4x v100 (only have 2 for now but planning on getting 2 more this winter) it would be pretty usable”

“We serve around 5 models (a mix of fine-tuned Llama models and a couple of embedding models) on one EKS cluster with a GPU node pool. Leadership now wants cost per model and eventually cost per 1M tokens so they can compare our infrastructure costs against just using an API. The problem is the GPU nodes are shared, some models sit idle half the day, and the bill just says “p4d node go brrr.” Namespace level cost feels too coarse, and idle capacity has to go somewhere. How are you handling this?”

“The command There isn't one. scripts/larg/ exposes train.sh , video logger.sh , bench.sh and deploy.sh ; just run <script only reaches scripts that live inside a workspace extension's scripts/ dir. What it lacks No way to run a one off analysis or eval script against a finished run's checkpoint on a leased GPU. This comes up at the end of every experiment: a census, a probe, a distribution dump — something that is not training, not play , and not a video. What the run needs After the SSTI reorie”

“I don't do training. The limitation is literally everything. Laptop GPUs have limited VRAM, limited bandwidth and limited compute compared to their desktop cointerparts. Thermals suck. System memory capacity and bandwidth sucks. Cost sucked even before prices went ballistic. No matter how you slice it, it's a waste of money”

“Hi everyone, thank you so much for all the interest in my model. It's more than I expected. Here is a short summary of the trial and error I went through while building Apex-2. 1. GPUs were always the bottleneck I planned to train on about 1T tokens, but in the end I could only train on about 80B. FineWeb-Edu alone is about 1.3T tokens, and I clearly underestimated the scale: a single H100 was not enough. This project really showed me why so much money goes into GPUs and VRAM. 2. DiLoCo With”

“Follow up from the 4421 review (non blocking finding 2, second half). Where: prosper/src/gpu/execute/gpu executor.cpp , the owned nested snapshot block in the ordered draw loop ( owned snapshot futile / flush span(true) ). Scenario: after any draw in a submit fails realization, graphics epoch ok stays false for the rest of that submit. Every later draw that wants the owned nested snapshot then gets producers complete == false , so its owned waves can never be admitted. It still pays the authorit”

“I mean not what would make you migrate but what would make you add another provider. I have heard all the usual reasons like Disaster recovery, Cost savings, Data residency, Customer demands, GPU availability, Regulatory needs, Avoiding vendor lock-in but I wanted to know which of these actually work when you try them in real life cause adding Azure, GCP, Yotta or OCI is not just about having another place to run your workloads. Now your team has to deal with another identity and access manageme”

I see. Thought it also supported that.

“Say one card in a 64-GPU job starts throwing Xid errors three hours in. The job dies with an NCCL timeout that doesn't tell you much, and somebody loses the afternoon going node by node to find the bad one. Do you test nodes before a big job starts, like DCGM diag or a quick NCCL all-reduce? Or just let it fail and pull the node afterwards? submitted by /u/Borges_Engineer [link] [comments]”

“Thunder Compute (YC S24) | C++ Systems, Infrastructure, BizOps | San Francisco (Onsite) | Full-time | https://www.thundercompute.comWe're building the VMware for GPUs. Most GPUs in the cloud sit idle a lot of the time because they're bolted to one machine over PCIe. We decouple them.Our virtualization layer is a userspace shim, loaded through LD_PRELOAD, that intercepts CUDA calls and ships them over the network to a host with a physical GPU somewhere else in the data center. Workloads run uncha”

“did you set it up so that only experts are on the ssd? (and ideally have the always active stuff in vram)? I expcet that if you use --n-cpu-moe to with max gpu offload to put all the always active in vram and only have experts in system ram and ssd you'd get more speed.”

“Clustering is the main benefit of the sparks. Otherwise you’ll have to choose a machine that doesn’t have enough ram, or the $120,000 variant dgx station and have nothing inbetween”

“You don't rely on them. You rely on open weight models on servers with ZDR you can trust.”

“Yes, there's a compute shortage. I'm squeezing my lab cluster with NVMe tiering to double density, but this isn't going to get better any time soon.”

“I’m not too experienced with this, but what is the reason no one has mentioned Azure as a solution ? Is it the cost ? My last job had a handful of VMs in azure . Unfortunately I don’t get much cloud expose in my new role.”

“Thanks for sharing this OP. If you are purchasing this for a company where running AI is seen as a support function, this cost is just small % of overall operating cost. Then there is the ROI factor, use it for runing use cases with greater ROI and owning could turn out more profitable. Sometimes, you can not rent because policies require data not to leave your premises. If you look at getting the increase in datacenter costs in terms of cooling, power, spares and support - renting could still b”

“for those who dont have the means to run locally, there's cloud subs/api. if you want to run custom models, there's gpu rental. Last time I looked at this you had to first rent a gpu from runpod/vast etc, storage (or use s3), manually connect, download and run the model, tools and finally get an inference endpoint you then use with a local client. Now I think this might be much simpler? eg HF can host your model, or there are other services like featherless. whats the process now and how”

just mine crypto instead

“Regarding our plans we actually calculated based on our operational costs for our servers and GPU resources we're utilizing, The costs were therefore determined by the range at which we actually make profit per group of users. I've reviewed Haggin and it does look ideal for Saas, but it gets different when your product utilizes resources that are expensive like GPU cloud resources therefore the stats determine the pricing model of the company”

“There's no world where this makes sense. Let's take it issue by issue.1. Hardware failures: If you know anything about data centers you know that servers fail all the time. I've seen estimates of AI DC servers that have a failure rate of about 9% annually. A terrestrial DC has hardware techs who get alerted to the failure, replace the bits with known good bits and then take the suspect bits for testing to either bin them or verify they're working. You have no option to economically do this in sp”

“API pricing is probably going to be cheaper than renting bare metal, and scales to your needs. The next best option, if you’re truly going to get your money’s worth - is to acquire the hardware yourself”

“Random thought I've been kicking around. We keep talking about GPU compute in terms of renting machines by the hour. But for a lot of SaaS AI workloads, the GPU itself isn't really what I want. I just want the job done. Say I have 500k documents to process. Why do I care whether that takes one GPU for 20 hours, 10 GPUs for 2 hours, or 100 GPUs for 12 minutes? What I actually care about is: $X + done by tomorrow. Obviously this gets complicated pretty quickly once you bring in privacy, re”

“I’d compare cost per completed task, including retries, rather than just GPU busy time. A cheaper setup stops being cheap if the model needs twice as many attempts to get something right”

“Requests were flat. Cost wasn't. I checked batching, model size, quantization, all the obvious things first, and all of them were fine. What's your debugging order when this happens? Feels like everyone has a different first move. submitted by /u/Born-Woodpecker-4530 [link] [comments]”

“I run a small service that measures what a recurring GPU job costs on rented cards, so read me as an interested party. Over three weeks we paid for 22 runs on RunPod, Vast.ai, Nebius and Hyperstack, about $37 in total. Here is what cost us time or money that no price list mentions. Startup is billed, and it varies a lot. RunPod containers were live in 5-10 seconds. The VMs on Nebius and Hyperstack took 3-7 minutes before anything ran. Nebius charged for those minutes, Hyperstack did not charge f”

“As some may already know, I made a Vulkan resident trainer so anyone with a gpu, can train their own model because almost all modern graphics cards can run Vulkan. But I came up with some problems. mainly that system ram would explode exponentially. so I fixed the memory leak, and also made a lot more of the work, go to GPU. The readme has all of the fixes in it now. If you still have questions, go ahead and ask them. If I can I'll answer them. submitted by /u/Savantskie1 [link]”

“No We’re partially running self hosted via a 4x rtx6000 Blackwell pro setup and since we’re training vision not a frontier llm training costs are way less than you’d expect. But in regards to like holding off to market until iOS and android are ready you wouldn’t? I just want to maximize the first retention of my user base especially before it gets too far or too many eyes on it”

“For Local deployment this idea could create opportunities for alternatives but i don't think corporations will buy cheaper hardware for their datacenter hoping they can just vibecode performance in the near future. Right now cuda still has more examples so even in this scenario Nvidia could benefit a bit from cuda because it will be easier for your local model to optimize inference on Nvidia hardware.”

“Keep the A40 as a staging box to validate model swaps before touching production. Failover sounds nice until you realize its weights have drifted from what you last deployed and nobody updated them. At concurrency 2 you won't need it running hot anyway.”

I love the box what is it I read noctua

“There is an error here when attempting to perform an operation on r (which is on device 1) while the current active device is 0.”

“yeah if the container and VM management utilities are good enough I suppose this might satisfy the Paperwork Gods”

“Vmware, cause migration is a 2 year project and would involve rebuilding workshops for multiple bussiness units, its in the pipeline but still a year or 2 away so till then we pay the tax”

“You want good locality when training. Distributed training over the internet with 2 Gbps connections with 5,000 microsecond latency would be excruciatingly slow. There is good reason nVidia worked on infiniband in the 1990s, bought Mellanox in 2020, and ships compute clusters with 400 Gbps networking at <1 microsecond latency between servers. If companies stop shipping open weight models we can crowd-source the training data, but I suspect the actual training will require a non-profit to rent”

“Ohh. Some people reported seeing no bill after downsizing and then going back to 4c/24gb. Not sure what OCI is doing. Check: https://www.reddit.com/r/oraclecloud/comments/1uynaqa/im_using_payg_and_can_confirm_4c_24g_is_still/”

“That is precisely what I was trying to do. To date I’ve not needed a gpu in my server as the onboard intel igpu has been enough. I guess I’ll check out the market on GPU’s then revisit this topic. I assume more vram for gpu is better? 16g enough?”

“It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire GPU server. There are managed options for LoRA serving on top of vLLM (AWS), but you generally still end up paying for an entire instance. I started wondering: if 99%+ of the model weights are identical between the base model and something like a rank 8–32 LoRA/QLoRA adapter, why does each adapter”

“I wrote an article about setting up local llm with multi GPU set up, with focus on budget options. Hope it helps newcomers here. submitted by /u/lblblllb [link] [comments]”

“What sort of work do you do? Not seen many individuals doing custom model development”

“yeah i see i can switch to luna on plus. i just want the extended memory for context. luna can handle the tasks i need done, once i up the ante on my game dev, i can hand off the more complex bits to sol”

“🚀 The feature, motivation and pitch I noticed that Modal (https://modal.com/blog/gpu mem snapshots) and InferX (https://inferx.net/) have implemented the CUDA checkpoint/restore API to drastically reduce cold start. I tried both of the services and it seems to work extremelly well. InferX told me they build it on top of vLLM so it definitly seems possible to do. Right now, I'm not aware of any open source implementation of this tech. I would love to see this feature implemented in vLLM as it wo”