AWS Machine Learning Blog· Dmitry Soldatkin·· 4 天前AI 评分58
Amazon SageMaker AI 实时端点支持 KV 缓存感知路由
Amazon SageMaker AI real-time endpoints now support KV cache aware routing
AI 导读
Amazon SageMaker AI 为实时端点新增 KV 缓存感知路由(KV_AWARE),根据模型容器上报的缓存状态把请求发到已缓存其提示词且负载较低的实例。
来源:AWS Machine Learning Blog · aws.amazon.com