gpt4all/gpt4all-backend
2023-09-12 14:16:02 -07:00
..
gptj mono repo structure 2023-05-01 15:45:23 -04:00
llama mono repo structure 2023-05-01 15:45:23 -04:00
llama.cpp-230511@f826aac617 move 230511 submodule to nomic fork, fix alibi assert 2023-06-30 21:07:21 -03:00
llama.cpp-230519@5ea4339273 Dlopen backend 5 (#779) 2023-05-31 17:04:01 -04:00
llama.cpp-mainline@9bee309a7c vulkan python bindings on windows fixes 2023-09-12 14:16:02 -07:00
scripts Bert 2023-07-13 14:21:46 -04:00
bert_impl.h Bert 2023-07-13 14:21:46 -04:00
bert.cpp bump llama.cpp version + needed fixes for that 2023-08-31 15:29:54 -04:00
CMakeLists.txt more circleci 2023-08-31 15:29:54 -04:00
dlhandle.h Fix for windows. 2023-08-31 15:29:54 -04:00
falcon_impl.h Bert 2023-07-13 14:21:46 -04:00
falcon.cpp bump llama.cpp version + needed fixes for that 2023-08-31 15:29:54 -04:00
gptj_impl.h Bert 2023-07-13 14:21:46 -04:00
gptj.cpp Add starcoder support. 2023-07-27 09:15:16 -04:00
llama.cpp.cmake Don't link against libvulkan. 2023-09-12 14:26:56 -04:00
llamamodel_impl.h Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
llamamodel.cpp Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
llmodel_c.cpp Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
llmodel_c.h Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
llmodel_shared.cpp Bert 2023-07-13 14:21:46 -04:00
llmodel_shared.h Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
llmodel.cpp Fix Windows unable to load models on older Windows builds 2023-08-09 09:27:43 +02:00
llmodel.h Nomic vulkan backend licensed under the Software for Open Models License (SOM), version 1.0. 2023-08-31 15:29:54 -04:00
mpt_impl.h Bert 2023-07-13 14:21:46 -04:00
mpt.cpp Use ggml scratch bufs for mpt and gptj models (#1104) 2023-06-30 10:53:45 -07:00
README.md Update README.md 2023-06-04 08:46:37 -04:00
replit_impl.h Bert 2023-07-13 14:21:46 -04:00
replit.cpp bump llama.cpp version + needed fixes for that 2023-08-31 15:29:54 -04:00
starcoder_impl.h Add starcoder support. 2023-07-27 09:15:16 -04:00
starcoder.cpp starcoder: use ggml_graph_plan 2023-08-31 15:29:54 -04:00
sysinfo.h Use sysinfo in backend. 2023-06-26 14:14:49 -04:00
utils.cpp non-llama: explicitly greedy sampling for temp<=0 (#901) 2023-06-08 11:08:30 -07:00
utils.h Use operator ""_MiB (#991) 2023-06-16 15:56:22 -04:00

GPT4ALL Backend

This directory contains the C/C++ model backend used by GPT4All for inference on the CPU. This backend acts as a universal library/wrapper for all models that the GPT4All ecosystem supports. Language bindings are built on top of this universal library. The native GPT4all Chat application directly uses this library for all inference.

What models are supported by the GPT4All ecosystem?

Currently, there are three different model architectures that are supported:

  1. GPTJ - Based off of the GPT-J architecture with examples found here
  2. LLAMA - Based off of the LLAMA architecture with examples found here
  3. MPT - Based off of Mosaic ML's MPT architecture with examples found here

Why so many different architectures? What differentiates them?

One of the major differences is license. Currently, the LLAMA based models are subject to a non-commercial license, whereas the GPTJ and MPT base models allow commercial usage. In the early advent of the recent explosion of activity in open source local models, the llama models have generally been seen as performing better, but that is changing quickly. Every week - even every day! - new models are released with some of the GPTJ and MPT models competitive in performance/quality with LLAMA. What's more, there are some very nice architectural innovations with the MPT models that could lead to new performance/quality gains.

How does GPT4All make these models available for CPU inference?

By leveraging the ggml library written by Georgi Gerganov and a growing community of developers. There are currently multiple different versions of this library. The original github repo can be found here, but the developer of the library has also created a LLAMA based version here. Currently, this backend is using the latter as a submodule.

Does that mean GPT4All is compatible with all llama.cpp models and vice versa?

Unfortunately, no for three reasons:

  1. The upstream llama.cpp project has introduced a compatibility breaking re-quantization method recently. This is a breaking change that renders all previous models (including the ones that GPT4All uses) inoperative with newer versions of llama.cpp since that change.
  2. The GPT4All backend has the llama.cpp submodule specifically pinned to a version prior to this breaking change.
  3. The GPT4All backend currently supports MPT based models as an added feature. Neither llama.cpp nor the original ggml repo support this architecture as of this writing, however efforts are underway to make MPT available in the ggml repo which you can follow here.

What is being done to make them more compatible?

A few things. Number one, we are maintaining compatibility with our current model zoo by way of the submodule pinning. However, we are also exploring how we can update to newer versions of llama.cpp without breaking our current models. This might involve an additional magic header check or it could possibly involve keeping the currently pinned submodule and also adding a new submodule with later changes and differienting them with namespaces or some other manner. Investigations continue.

What about GPU inference?

In newer versions of llama.cpp, there has been some added support for NVIDIA GPU's for inference. We're investigating how to incorporate this into our downloadable installers.

Ok, so bottom line... how do I make my model on Hugging Face compatible with GPT4All ecosystem right now?

  1. Check to make sure the Hugging Face model is available in one of our three supported architectures
  2. If it is, then you can use the conversion script inside of our pinned llama.cpp submodule for GPTJ and LLAMA based models
  3. Or if your model is an MPT model you can use the conversion script located directly in this backend directory under the scripts subdirectory

Check back for updates as we'll try to keep this updated as things change!