actually the math is more like $1500 a month on a MI300x on a highly optimized rocm/sglang image and everyone's getting 100+ tok/sec and we have 100 active people and plenty of room for others.
but were losing money right now obviously.
we did come from 4x 3090s could actually make money, but we thought a couple hundred bucks a month isnt worth a 100 degree office.
People are dumping 128k cold documents in like its their job. Some dudes hit 2 billion + tokens a week.
i know, but im scared to own our own auth, when there's money involved... too cheap to pay for a service. So we lean on the back of the giants for now.
If it helps, there's really no reason to ever log into our site after you get an API Key. I guess until you want to make another api key
So on mi 300x we run FP8 and that was original plan.
We are doing some weird q4 on 3090s now that is surprisingly good (for qwen) .
will people pay for it? Yeah, sometimes. Some dude just burned 80mil tokens in 24 hr and posted it to our discord. I'd say he's getting 600 pennies worth.
I think the best part is that no one is watching what your doing, you don't worry, and it's fun.
Also just as divided as people are to whether it's a good deal or not, people are divided on liking/hating the name and brand.