Ask HN: Anyone have dedicated, local, AI server?
1 comments
Do you have a spare computer that you can load up with RAM so that you can try out some available models? It may not be fast, but it will give you an idea of how capable the models are.
I have been running some local models on LMStudio using an AMD 5950XT with 128 GB Ram (until recently with no GPU offloading). My two cents is that I keep reaching for my free ChatCPT account because I find the local models fairly unreliable.
Some just seem to get into a thinking loop where they go over the same ground again and again. Others can just output pure garbage.
Would anyone here suggest any specific models. Especially for Swift/SwiftUI. PHP. And MySQL Stored Procedures.
I have been running some local models on LMStudio using an AMD 5950XT with 128 GB Ram (until recently with no GPU offloading). My two cents is that I keep reaching for my free ChatCPT account because I find the local models fairly unreliable.
Some just seem to get into a thinking loop where they go over the same ground again and again. Others can just output pure garbage.
Would anyone here suggest any specific models. Especially for Swift/SwiftUI. PHP. And MySQL Stored Procedures.
I'm not, yet, looking into training models, but more about inference, while staying relatively on a budget. I have about €2-3k to spend on hardware, for a server I can run at home. I know that one can rent an AI inference server, but from my research, they tend to cost around €200-300/mo, which means that my hardware will pay for itself in ~10 months.
However, I'm quiet confused with what hardware to chose. I saw recommendations for Ryzen AI Max+ 395, in SFF PCs, but some people suggest instead to build a "regular" PC with Nvidia 3090 or 4090.
People who run small servers (as opposed to €10-15k beasts), what's your experience and recommendations?