text-generation-webui/docs/llama.cpp-models.md

## Using llama.cpp in the web UI

#### Pre-converted models

Place the model in the `models` folder, making sure that its name contains `ggml` somewhere and ends in `.bin`.

#### Convert LLaMA yourself

Follow the instructions in the llama.cpp README to generate the `ggml-model-q4_0.bin` file: https://github.com/ggerganov/llama.cpp#usage

## Performance

This was the performance of llama-7b int4 on my i5-12400F:

> Output generated in 33.07 seconds (6.05 tokens/s, 200 tokens, context 17)

You can change the number of threads with `--threads N`.
Add files via upload 2023-04-22 01:34:13 -04:00			`## Using llama.cpp in the web UI`

Don't require llama.cpp models to be placed in subfolders 2023-04-22 13:56:48 -04:00			`#### Pre-converted models`
Add files via upload 2023-04-22 01:34:13 -04:00
Update llama.cpp-models.md 2023-05-11 08:47:36 -04:00			Place the model in the `models` folder, making sure that its name contains `ggml` somewhere and ends in `.bin`.
Add files via upload 2023-04-22 01:34:13 -04:00
Don't require llama.cpp models to be placed in subfolders 2023-04-22 13:56:48 -04:00			`#### Convert LLaMA yourself`
Add files via upload 2023-04-22 01:34:13 -04:00
Don't require llama.cpp models to be placed in subfolders 2023-04-22 13:56:48 -04:00			Follow the instructions in the llama.cpp README to generate the `ggml-model-q4_0.bin` file: https://github.com/ggerganov/llama.cpp#usage
Add files via upload 2023-04-22 01:34:13 -04:00
			`## Performance`

			`This was the performance of llama-7b int4 on my i5-12400F:`

			`> Output generated in 33.07 seconds (6.05 tokens/s, 200 tokens, context 17)`

Don't require llama.cpp models to be placed in subfolders 2023-04-22 13:56:48 -04:00			You can change the number of threads with `--threads N`.