From 40d3d139968d46721f5617e6a367d921017fa572 Mon Sep 17 00:00:00 2001 From: Xuechen Li Date: Wed, 15 Mar 2023 09:35:19 -0700 Subject: [PATCH] fix typo in weight decay; clarify python version. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 114498c..f5b8c52 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ We fine-tune our models using standard Hugging Face training code with the follo | Learning rate | 2e-5 | | Epochs | 3 | | Max length | 512 | - | Weight decay | 1 | + | Weight decay | 0 | Given Hugging Face hasn't officially supported the LLaMA models, we fine-tuned LLaMA with Hugging Face's transformers library by installing it from a particular fork (i.e. this [PR](https://github.com/huggingface/transformers/pull/21955) to be merged). The hash of the specific commit we installed was `68d640f7c368bcaaaecfc678f11908ebbd3d6176`. @@ -111,6 +111,7 @@ pip install -r requirements.txt Then, install the particular fork of Hugging Face's transformers library. Below is a command that fine-tunes LLaMA-7B with our dataset on a machine with 4 A100 80G GPUs in FSDP `full_shard` mode. +We were able to reproduce a model of similar quality as the one we hosted in our demo with the following command using **Python 3.10**. Replace `` with a port of your own, `` with the path to your converted checkpoint and tokenizer (following instructions in the PR), and `` with where you want to store your outputs.