11 Commits

Author	SHA1	Message	Date
Adam Treat	7f9f91ad94	Revert "New tokenizer implementation for MPT and GPT-J" ... This reverts commit bbcee1ced5.	2023-05-30 12:59:00 -04:00
Aaron Miller	bbcee1ced5	New tokenizer implementation for MPT and GPT-J ... Improves output quality by making these tokenizers more closely match the behavior of the huggingface `tokenizers` based BPE tokenizers these models were trained with. Featuring: * Fixed unicode handling (via ICU) * Fixed BPE token merge handling * Complete added vocabulary handling	2023-05-30 12:05:57 -04:00
Adam Treat	9bfff8bfcb	Add new reverse prompt for new localdocs context feature.	2023-05-25 11:28:06 -04:00
Juuso Alasuutari	81fdc28e58	llmodel: constify LLModel::threadCount()	2023-05-22 08:54:46 -04:00
aaron miller	e6fd0a240d	backend: fix buffer overrun in repeat penalty code ... Caught with AddressSanitizer running a basic prompt test against llmodel standalone. This fix allows ASan builds to complete a simple prompt without illegal accesses but there are still notably several leaks.	2023-05-17 07:54:10 -04:00
kuvaus	507e913faf	gpt4all-backend: Add MSVC support to backend (#595) ... * Add MSVC compatibility * Add _MSC_VER macro --------- Co-authored-by: kuvaus <kuvaus@users.noreply.github.com>	2023-05-16 11:35:33 -04:00
Aaron Miller	d14936bfd6	backend: dedupe tokenizing code in mpt/gptj	2023-05-16 10:30:19 -04:00
Aaron Miller	6182026c70	backend: dedupe tokenizing code in gptj/mpt	2023-05-16 10:30:19 -04:00
Aaron Miller	4cd8bdf9a1	backend: make initial buf_size const in model impls ... more unifying mpt and gptj code - this one's never written so also changing the name to be clearer	2023-05-16 10:30:19 -04:00
Aaron Miller	08402a1b64	mpt: use buf in model struct (thread safety)	2023-05-16 10:30:19 -04:00
Adam Treat	d918b02c29	Move the llmodel C API to new top-level directory and version it.	2023-05-10 11:46:40 -04:00