FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper GPUs
In brief
- CUDA-optimized implementation for Llama-style Transformers on Hopper GPUs.
- FP8 prefill and INT4 decoding with up to 2.18x speedup vs PyTorch.
- Perplexity reduced from 7.23 to 6.99 with GPTQ in INT4 format.
FuTransformer introduces a CUDA-optimized implementation for Llama-style Transformers on NVIDIA Hopper GPUs. It combines quantization techniques (FP8 for prefill, INT4 for decoding) and hardware-aware optimizations to accelerate inference, achieving up to 2.18x speedups over existing solutions.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error