Engineering is not running a business. Different skill set.. you can learn it, but a side business skill needs marketing, operations, etc.. even a small one. You could make the best mousetrap in the world, but if no one knows about it, it goes no where. I ran into this many times.. great idea, could solve a lot of problems, but could never figure out how to market it.
LLMs are just a tool. Doesnt mean you dont have to think. They lie all the time, take short cuts, try to please you, etc. I treat them like a consultant almost, and make them explain what they are doing and why all the time. They can do things i can never do, but often they just do things that I could do, just faster. Yes i could code that, but an LLM can literally type the code faster than i ever could. I also find that plan mode makes me think.. just my $0.02
when i hire people, i look for one thing.. are they smart... period.. end of story. if they pass what i think is smart ( and yes its subjective), the you start to drill down into experience, personality, etc. Smart people apply what they have learned to a variety of skills, and are willing to learn more. You can find them anywhere.. one of the best hires i ever found was in an ice cream shop.. she became one of the best PMs i ever had.. We put so many people into silos, sometimes the solution is staring you right in the face!!
honestly it's mostly gpu supply. scaling up to handle load means spinning up new nodes, and that takes minutes not seconds because the models are huge and need multiple coordinated gpus per instance.
also worth saying, even when things are "up" you often get different answers to the same question. that's the reliability problem nobody talks about. fine for a chatbot, not fine if you're building anything that needs to be repeatable and deterministic... i moved more to the ML route, but i guess it depends on what you are trying to do.
love the budget-constraints section, feels way too familiar. i'm running 7 NLP models on an $11/month VPS for a different project and every architecture decision ends up being "what's the cheapest way that doesn't burn my bank account on one runaway request."
question — how are you handling demucs cold starts on modal? cold start was what eventually pushed me off modal for a request-response
use case. the user is staring at a spinner for 20-30s on first invocation and it kills conversion. did you solve it or just eat it because the rest is so computationally heavy anyway?