I already had 2x 5070's that I had purchased a year or more ago, so getting an additional two to fill up the PCIe slots on the motherboard seemed reasonable.
I did some math/shopping as well. To get 48GB of VRAM you can get 2x 3090s but that is $3k. A single 5090 is $4k but has 32GB, great for running models like Qwen 27B but maybe nothing else depending on your model settings. Already having 2x 5070, where each card is around $600, it made sense for me to get two more which was $1200 and the memory speeds aligned.
The best value option if you're building from scratch is go with 5060 ti (16GB VRAM). Each of those cards are $570/each on Amazon, cheaper than 4x 5070's. Only downside is memory speed is slightly slower, but you wind up with 64GB of VRAM and you can run big models and small models alongside each other comfortably.
In my setup I ran Qwen3.5 9B for fast inference on simple things and Qwen3.6 27B Q6 for coding work. But I ran into stability issues, so I use llama-swap to dynamically swap models. But with 64GB of VRAM, you wouldn't have that issue. There is overhead to loading LLMs into VRAM that isn't clear, so having extra VRAM is a helpful buffer.
I'm using 4x RTX 5070's and first-gen AMD threadripper (1950X) to run Qwen3.6 27B (MTP) Q6_K with llama.cpp and it works great as a daily driver with Pi. Around 50-60 toks/sec. I also connect a few other applications to it such as OpenWeb UI and recently set up Bifrost, an LLM gateway, to be the primary access point for the models I serve.
I've tried other models such as Qwen3.6 35B A3B and I've found that 27B works better for me when it comes to coding. It's slower being a dense model but the quality seems much better. Inference on my system for Qwen3.6 35B A3B is around 130-140 toks/sec, non-MTP, which is insanely fast!
You don't need 4x 5070's to run Qwen3.6 27B, three or maybe even two will work. However, I use MTP (multi-token prediction) to speed up 27B and that eats up more memory because the draft model requires its own context.
Another thing to keep in mind is that the tools you're using have their system prompts that are loaded into the model for each conversation. When I fire up Pi, working with the model is very snappy at start. When I interact with the LLM via Hermes CLI, it's much slower. That's because each prompt with Hermes is loading so much stuff (skills, tools, etc.) into the context and then it's there forever until the conversation ends.
I like running models at home for privacy, but I also like how there are no quotas, usage isn't a worry. If the future is "loop engineering" then you will be burning through tokens and $$$ using a cloud models.
My system idles around 200W and is around 350-450W when inference load is high. Decoding (token generation) isn't all that efficient, and your GPUs sit idle more than you think during inference. Advancements like diffusion may 1) speed up decoding and 2) let you utilize more of your idle GPU.
I run Packetriot: https://packetriot.com, which is an alternative to ngrok. It's been in operation for 6-ish years and reached $500/month in year two. It doesn't replace my salary (yet) but that's what I'm working toward. Thankful to have thousands of users and some big companies using our services or self-hosting our server.
This is a fun space to work in but there's lot of small competitors and opensource alternatives. In the first few years it was sometimes demotivating, but at some point I began seeing competitors bow out and Packetriot kept getting better and better each year.
This year I published a lot of updates across the platform. Our UX is better with a new web-based UI for our client and all of the features of the platform can be managed in the client UI. No more going back forth between the client and user portal.
I'm planning on releasing a community edition of our server in 2026 so that anyone can use our network server for free, for personal use or to evaluate for commercial use.
Our services were used for C2 as well. I investigated it a bit but eventually decided to just drop TCP forwarding from our free-tier and that reduced our abuse/malware reports for C2 over TCP to zero essentially.
One path I looked at was to use the VirusTotal API to help identify C2's that other security organizations were identifying and leverage that to automatically take down malicious TCP endpoints. I wrote some POCs but did not deploy them. It's something I plan on taking up again at some point next year.
Thanks for sharing this. I run packetriot.com, another tunneling service and I ended up writing my own scanner for endpoints using keyword lists I gathered from various infosec resources.
I had done some account filtering for origins coming out of Tor, VPN networks, data centers, etc. but I recently dropped those and added an portal page for free accounts, similar to what ngrok does.
It was very effective at preventing abuse. I also added mechanism for reporting abuse on the safety page that's presented.
Similar to ngrok with our own differences and approach. I also publish another product called Spokes Gateway which builds on the tunneling server and includes support for service meshes, high-availability, clusters and some other features.
I'm building a separate website for Spokes and its related software, hoping to publish it soon. It's eventual home will be https://spokes.network.
I did some math/shopping as well. To get 48GB of VRAM you can get 2x 3090s but that is $3k. A single 5090 is $4k but has 32GB, great for running models like Qwen 27B but maybe nothing else depending on your model settings. Already having 2x 5070, where each card is around $600, it made sense for me to get two more which was $1200 and the memory speeds aligned.
The best value option if you're building from scratch is go with 5060 ti (16GB VRAM). Each of those cards are $570/each on Amazon, cheaper than 4x 5070's. Only downside is memory speed is slightly slower, but you wind up with 64GB of VRAM and you can run big models and small models alongside each other comfortably.
In my setup I ran Qwen3.5 9B for fast inference on simple things and Qwen3.6 27B Q6 for coding work. But I ran into stability issues, so I use llama-swap to dynamically swap models. But with 64GB of VRAM, you wouldn't have that issue. There is overhead to loading LLMs into VRAM that isn't clear, so having extra VRAM is a helpful buffer.