Major upgrades to Ray Serve: 88% lower latency and 11.1x higher throughput(anyscale.com)

Major upgrades to Ray Serve: 88% lower latency and 11.1x higher throughput

https://www.anyscale.com/blog/ray-serve-inference-lower-latency-higher-throughput-haproxy

1 コメント

djwjjtw4 か月前

The real story here is replacing Ray Core's actor task dispatch with direct gRPC for the hot path. Essentially admitting that the general-purpose actor model was the bottleneck for inference workloads. Smart move — let Ray handle orchestration and get out of the way for actual data flow.