AI · Developers · Stable
Dependence on cloud hosted AI solutions due to high costs and limitations of local hardware
Users struggle with the high costs associated with cloud hosted AI services and seek alternatives that can be run locally.
Who experiences it: Developers
Momentum
0%
Pain
75
Competition
90
Opportunity
63/100
Signals over time
6 observed signals across 1 sources, tracked for 47 days. Confidence: low.
What people are saying Observed
“Your analysis is correct if you’re maxing out the $20 Claude subscription and using the best models to do things that need high quality results. In my opinion, what we actually have to compare against is the $20 subscription compared to what using Claude or ChatGPT is like with the free plan. If I use Claude free it cuts me off from using it for hours after a very small number of questions being asked, and it can’t be hooked up to a coding system at all (no Claude code CLI or VSCode extension). So when I dropped a $500 GPU into my gaming server in my closet, I’m breaking even compared to the $20/month Claude subscription in 2 years (and my GPU already appreciated in value since I bought it). I can send as many queries as I want in any way I want, including custom applications that would have normally been billed at API token rates. On top of that, all my data is kept private to my server. (I’m not counting electricity because my server was already on serving all my self-hosted apps, so a significant amount of electricity is only being added to the load when I’m actively asking questions)”
Hacker News · frustration
“It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point. In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though. If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users. At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actuall”
Hacker News · frustration
“> AI will end up being just slightly more expensive than the raw electricity required to produce it. AFAICT, this hasn't happened for any "cloud" compute company that has been in business for -say- at least five years. Unless your prediction is that roughly no one will use "cloud" "AI", I'm rating this prediction as "totally bogus". And if that is your prediction, then what you're saying is Assuming that you're not doing anything that needs exotic system administrators, nearly all of the cost of computing when you own your own hardware is the power for that hardware. which... unless you're buying used hardware, I don't think that's true, either. Hetzner quotes a max of 167.23 EUR/month for a full rack and what I think is 2400W of power, and lists the rate for Finnish power as 0.22 EUR per kWh. [0] If assume that Hetzner's spending cap doesn't apply and we are forced to pay the full quoted rate for power in Finland, I think that works out to a bit under 4.6K EUR per year: 240V * 10A = 2.4kW 2.4kW * 24H * 30 * 12 = 20,736 kWh 20,736kWh * 0.22 EUR/kWh = 4562 EUR/year The last time I ”
Hacker News · complaint
“Is anyone self-hosting their own AI?. submitted by StateMachine to selfhosted 1 points | 0 comments Does anyone run their own AI service? If so, Which one? What hardware? Do you have a way to access it from outside or from a phone? And, if you’re open to sharing, what do you do with it? I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes. I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.”
RSS Feeds · frustration
“My GPUS are Rtx3090s with 24GB ram. You guessed correctly. I do not enjoy the fact those cards cost more than their msrp 6 years later, but there is a much more important consideration than money (which also makes sense, but about that later). It is the fact soon people will not be able to do my job without AI at all. Even now if I didn't use it I think I'd be out competed very quickly. And having the ability to run it locally, using a really useful, not toy model is very useful. It makes you independent from Anthropic deciding to ban your account for example. As for money, it is an open secret the biggest cost of coding agents use is input tokens not generation. I tend to use about 1.3B input tokens per week on claude code with only 7-8M out. Out if this 80% is cached. And the cache is pretty restrictive. You have 5min cache and 1h cache. If you don't keep reading over that time your cache expires on the cloud. Then your 500k context counts as 500k input in its entirety. And the numbers I mentioned would cost thousands of USD a week at API prices. But when you control inference, you can keep your cache for as long as you want and save it to disk. I tend to have up t”
Hacker News · frustration
Why now? AI inference
Existing solutions Observed
- Apache MXNet · Free · complaints: Less popular than TensorFlow and PyTorch, leading to fewer resources, Documentation can be lacking, Steeper learning curve for some users
- TensorFlow · Free · complaints: Steep learning curve for beginners, Can be complex to set up for certain applications, Performance can vary based on hardware
- PyTorch · Free · complaints: Less mature than TensorFlow in some areas, Limited deployment options compared to TensorFlow, Can be resource-intensive
- Hugging Face Transformers · Free (with paid options for hosting) · complaints: Resource-intensive for local deployment, Limited support for non-NLP tasks, Some models may require significant tuning
- ONNX Runtime · Free · complaints: Limited documentation compared to other frameworks, Some features may not be fully supported, Can be challenging to troubleshoot
There is a gap for a cost-effective, locally deployable AI solution that can match the performance and flexibility of cloud-hosted options while addressing the limitations of existing frameworks. Current competitors primarily focus on cloud solutions or have significant resource demands for local deployment, leaving developers seeking a more efficient alternative.
See the full evidence, competitor gap matrix and opportunity report.
Free account. No credit card.