AI/gpt4all
1
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all.git synced 2024-10-01 01:06:10 -04:00
Code Issues Packages Projects Releases Wiki Activity

Revert "add tokenizer readme w/ instructions for convert script"

Browse Source 
This reverts commit 5063c2c1b2.
This commit is contained in:
Adam Treat 2023-05-30 12:58:18 -04:00
parent 5063c2c1b2
commit b5edaa2656
1 changed files with 0 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
gpt4all-backend/tokenizer/README.md
View File

@ -1,14 +0,0 @@
# BPE tokenizer
This is a C++ implementation of the encoding/decoding functions of a pretrained GPT-2 style BPE tokenizer. It is meant to be compatible with the GPT-J and MPT-7B tokenizers that were trained with HuggingFace [`tokenizers`](https://github.com/huggingface/tokenizers), and only implements the necessary functionality for those models (it is assumed that strings should always be [normalized](https://en.wikipedia.org/wiki/Unicode_equivalence) to Unicode NFC form and split with the GPT-2 "pretokenizing" [regular expression](https://github.com/karpathy/minGPT/blob/37baab71b9abea1b76ab957409a1cc2fbfba8a26/mingpt/bpe.py#L92))
## Converting a tokenizer file
`scripts/gen_tokenizer_include.py` can be used to convert a huggingface `tokenizers` `tokenizer.json` file into a C++ header file:
```bash
# get tokenizer.json
cd gpt4all-backend
wget -O /tmp/gptj-tokenizer.json https://huggingface.co/nomic-ai/gpt4all-j/raw/main/tokenizer.json
python ./scripts/gen_tokenizer_include.py /tmp/gptj-tokenizer.json gptj > ./tokenizer/gptj_tokenizer_config.h
```