CPU-GPU collaborative MoE inference system for efficient execution
In brief
- Adaptive method dynamically allocates MoE experts between CPU and GPU based on their load.
- Reduces PCIe transfers by targeting only high-load experts for GPU execution.
- Delivers up to 51.9% faster prefill speed and 39.5% higher decoding throughput.
A new inference method for MoE models optimizes dynamic expert allocation between CPU and GPU. It reduces latency by transferring only high-load experts to the GPU while running others on the CPU. Performance gains reach up to 51.9% in prefill speed and 39.5% in decoding throughput.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error