Skip to main navigation Skip to search Skip to main content

TPI-LLM: Serving 70B-Scale LLMs Efficiently on Low-Resource Mobile Devices

  • Zonghang Li
  • , Wenjiao Feng
  • , Mohsen Guizani*
  • , Hongfang Yu
  • *Corresponding author for this work
  • Mohamed Bin Zayed University of Artificial Intelligence
  • University of Electronic Science and Technology of China

Research output: Contribution to journal › Article › peer-review

13 Scopus citations

Abstract

LLM serving is shifting from cloud to edge due to privacy concerns over user interaction data. However, mobile devices struggle with very limited computing power and memory, requiring collaboration among multiple devices to run LLM apps. The mainstream solution, pipeline parallelism, is inefficient for such cases because mobile devices typically run only one inference task at a time. This article argues that tensor parallelism, despite its high communication cost, can better fit such scenarios. We introduce TPI-LLM, a compute and memory-efficient tensor parallel inference system designed to run 70B-scale LLMs on low-resource mobile devices. It keeps sensitive raw data local on users’ devices and employs a sliding window memory scheduler to dynamically manage layer weights. It overlaps disk I/O with computation and communication, enabling efficient operation of large models on memory-limited devices. Extensive experiments show that TPI-LLM reduces token latency by 80%–90% compared to Transformers, Accelerate, and Galaxy. It also cuts the peak memory footprint by 90%, requiring just 3.1 GiB of memory for 70B-scale models.

Original languageEnglish
Pages (from-to)3321-3333
Number of pages13
JournalIEEE Transactions on Services Computing
Volume18
Issue number5
DOIs
StatePublished - 2025
Externally publishedYes

Keywords

  • Distributed Inference
  • Edge AI
  • Large Language Model

Fingerprint

Dive into the research topics of 'TPI-LLM: Serving 70B-Scale LLMs Efficiently on Low-Resource Mobile Devices'. Together they form a unique fingerprint.

Cite this