2023-10-21 18:15:54 -04:00
## What Works
| Loader | Loading 1 LoRA | Loading 2 or more LoRAs | Training LoRAs | Multimodal extension | Perplexity evaluation |
|----------------|----------------|-------------------------|----------------|----------------------|-----------------------|
2024-02-04 21:48:04 -05:00
| Transformers | ✅ | ✅\*\*\* | ✅\* | ✅ | ✅ |
| llama.cpp | ❌ | ❌ | ❌ | ❌ | use llamacpp_HF |
| llamacpp_HF | ❌ | ❌ | ❌ | ❌ | ✅ |
2023-10-21 18:15:54 -04:00
| ExLlamav2_HF | ✅ | ✅ | ❌ | ❌ | ✅ |
2024-02-06 09:26:27 -05:00
| ExLlamav2 | ✅ | ✅ | ❌ | ❌ | use ExLlamav2_HF |
2023-10-21 18:15:54 -04:00
| AutoGPTQ | ✅ | ❌ | ❌ | ✅ | ✅ |
2024-02-04 21:48:04 -05:00
| AutoAWQ | ? | ❌ | ? | ? | ✅ |
| GPTQ-for-LLaMa | ✅\*\* | ✅\*\*\* | ✅ | ✅ | ✅ |
2023-10-21 18:15:54 -04:00
| ctransformers | ❌ | ❌ | ❌ | ❌ | ❌ |
2024-02-04 21:48:04 -05:00
| QuIP# | ? | ? | ? | ? | ✅ |
| HQQ | ? | ? | ? | ? | ✅ |
2023-10-21 18:15:54 -04:00
❌ = not implemented
✅ = implemented
\* Training LoRAs with GPTQ models also works with the Transformers loader. Make sure to check "auto-devices" and "disable_exllama" before loading the model.
\*\* Requires the monkey-patch. The instructions can be found [here ](https://github.com/oobabooga/text-generation-webui/wiki/08-%E2%80%90-Additional-Tips#using-loras-with-gptq-for-llama ).
2023-11-19 10:57:16 -05:00
\*\*\* Multi-LoRA in PEFT is tricky and the current implementation does not work reliably in all cases.
