From ba58e3b0b35e99d0b0889eb23e187bd018ed6863 Mon Sep 17 00:00:00 2001 From: James Ravenscroft Date: Sun, 23 Apr 2023 08:38:17 +0100 Subject: [PATCH] add build instructions for blas and cublas --- BUILD.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/BUILD.md b/BUILD.md index 56e2439..19d1ac9 100644 --- a/BUILD.md +++ b/BUILD.md @@ -50,3 +50,26 @@ Where: - *codegen* is a command line tool for testing out prompts in a lightweight way (a lot like llama.cpp) - *codegen-serve* is the actual REST server that can be used to connect to VSCode - *codegen-quantize* is the tool for quantizing models exported by the conversion script. For more details see [Converting and Quantizing The Models](https://github.com/ravenscroftj/turbopilot/wiki/Converting-and-Quantizing-The-Models). + +### Building with OpenBLAS + + +[BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) libraries accelerate mathematical operations. You can use the OpenBLAS implementation with Turbopilot to make generation faster - particularly for longer prompts. + +When you run cmake, you can additionally set `-D GGML_OPENBLAS=On` to enable BLAS support. + +E.g. `cmake .. -D GGML_OPENBLAS=On` + +### Building with CuBLAS + +CuBLAS is the [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) library provided by nvidia that runs linear algebra code on your GPU. This can speed up the application significantly, especially when working with long prompts. + +#### Install Cuda SDK for your Operating System + +You will need `nvcc` and the `libcublas-dev` dependencies as a bare minimum. Follow the guide from nvidia [here](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) for more detailed installation instructions. + +#### Configuring Cmake with CuBLAS + +You will need to set `-DGGML_CUBLAS=ON` and also pass the path to your `nvcc` executable with `-DCMAKE_CUDA_COMPILER=/path/to/nvcc`. + +Full example: `cmake -DGGML_CUBLAS=ON -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc ..`