AI/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-10-01 01:26:03 -04:00
Code Issues Packages Projects Releases Wiki Activity
9f40032d32
text-generation-webui/docs/ExLlama.md
oobabooga 9f40032d32
Add ExLlama support (#2444)
2023-06-16 20:35:38 -03:00

465 B
Raw Blame History

ExLlama

About

ExLlama is an extremely optimized GPTQ backend for LLaMA models. It features much lower VRAM usage and much higher speeds due to not relying on unoptimized transformers code.

Installation:

  1. Clone the ExLlama repository into your repositories folder:
cd repositories
git clone https://github.com/turboderp/exllama
  1. Follow the remaining set up instructions in the official README: https://github.com/turboderp/exllama#exllama