Petals – Run LLMs at home, BitTorrent-style
- Run large language models at home, BitTorrent‑style Generate text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+) or BLOOM (176B) and fine‑tune them for your tasks — using a consumer-grade GPU or Google Colab.
- You load a part of the model, then join a network of people serving its other parts.
- Single‑batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Unverified
- Run large language models at home, BitTorrent‑style Generate text with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+) or BLOOM (176B) and fine‑tune them for your tasks — using a consumer-grade GPU or Google Colab.
- You load a part of the model, then join a network of people serving its other parts.
- Single‑batch inference runs at up to 6 tokens/sec for Llama 2 (70B) and up to 4 tokens/sec for Falcon (180B) — enough for chatbots and interactive apps.
Sources: Petals