> I know multiple startups that use LLMs as their core bread-and-butter intelligence platform instead of tuned but traditional NLP models
It seems like LLMs would be perfect for start-ups that are iterating quickly. As the business, problem, and data mature though I would expect those LLMs to be consolidated into simpler models. This makes sense from a cost and reliability perspective. I wonder also about the impact of making your core IP a set of prompts beholden to the behavior of someone else’s model.
GPU RAM quantity isn’t typically correlated to inference rate. Precision/quantization levels do affect model size, which will affect inference rate. However, I would expect a smaller model to be faster (less RAM).
It seems like LLMs would be perfect for start-ups that are iterating quickly. As the business, problem, and data mature though I would expect those LLMs to be consolidated into simpler models. This makes sense from a cost and reliability perspective. I wonder also about the impact of making your core IP a set of prompts beholden to the behavior of someone else’s model.