The CPU is back: Rethinking the CPU-GPU split for LLM inference
- For the past 3 years, graphics processing units (GPUs) have dominated the large language model (LLM) conversation.
- In traditional chatbot applications, central processing units (CPUs) provide a fraction of the total compute per request, while GPUs do the heavy lifting.
- However, inference isn't a single model answering a single question.
Unverified
- For the past 3 years, graphics processing units (GPUs) have dominated the large language model (LLM) conversation.
- In traditional chatbot applications, central processing units (CPUs) provide a fraction of the total compute per request, while GPUs do the heavy lifting.
- However, inference isn't a single model answering a single question.
Sources: Redhat