Latency-SLO-Aware Memory Offloading for Large Language Model Inference

Authors

• Chenxiang Ma, Peking University
• Hanyu Zhao, Alibaba Group Holding Limited
• Zhisheng Ye, Peking University
• Zehua Yang, Peking University
• Tianhao Fu, Peking University
• Jiaxun Han, Peking University
• Jie Zhang, Peking University
• Yingwei Luo, Peking University
• Xiaolin Wang, Peking University
• Zhenlin Wang, Michigan Technological UniversityFollow
• Yong Li, Alibaba Group Holding Limited
• Diyu Zhou, Peking University

Document Type

Conference Proceeding

Publication Date

7-5-2026

Department

Department of Computer Science

Abstract

Offloading large language models (LLMs) states to host memory during inference promises to reduce operational costs by supporting larger models, longer prompts, and larger batch sizes. However, the design of existing memory offloading mechanisms does not take latency service-level objectives (SLOs) into consideration. As a result, they either lead to frequent SLO violations or underutilize host memory, thereby incurring economic loss and thus defeating the purpose of memory offloading. This paper presents NovaServe, a latency-SLO-aware memory offloading system for LLM serving. A key challenge in designing NovaServe is to reconcile the tension between meeting SLOs and maximizing host memory usage. To address this, NovaServe introduces a dual-dimensional offloading strategy that jointly controls the weight offloading ratio and attention computation placement to capture the precise tradeoff between SLOs and host memory usage. With that, NovaServe proposes a two-stage approach to automatically pick the strategy. The first stage is offline which generates the optimal offloading ratios for certain SLOs, while the second stage adjusts the strategy at the granularity of inference iteration based on runtime hardware status. Our evaluation shows that NovaServe consistently meets SLOs and improves the serving throughput over existing mechanisms by 2.3 × through efficiently utilizing host memory.

Publisher's Statement

© 2026 Copyright held by the owner/author(s). Publisher’s version of record: https://doi.org/10.1145/3797905.3807874

Publication Title

Proceedings of the International Conference on Supercomputing

ISBN

9798400725227

Recommended Citation

Ma, C., Zhao, H., Ye, Z., Yang, Z., Fu, T., Han, J., Zhang, J., Luo, Y., Wang, X., Wang, Z., Li, Y., & Zhou, D. (2026). Latency-SLO-Aware Memory Offloading for Large Language Model Inference. Proceedings of the International Conference on Supercomputing, PartF226351, 919-932. http://doi.org/10.1145/3797905.3807874
Retrieved from: https://digitalcommons.mtu.edu/michigantech-p2/2913

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-No Derivative Works 4.0 International License.

Version

Publisher's PDF