I think this the difference between compute bound pre-fill (a cpu has a high bandwidth/compute ratio), vs decode.
The time to first token is below 0.5s - even for a 10k context.
We wrote a rust py03 client for OpenAI embeddings compatible servers (openai.com, or infinity, TEI, vllm, sglang).
Most server-side ML infrastructure auto-scales based on the workload. On embedding workloads, this is no longer the bottleneck and has shifted to the client. In Python, the client is blocked by the global interpreter lock.
With the performance package, we release the gil during requests, so you have available resources to query your VectorDB again.