AI/alpaca-lora
1
0
Fork 0
mirror of https://github.com/tloen/alpaca-lora.git synced 2024-10-01 01:05:56 -04:00
Code Issues Packages Projects Releases Wiki Activity

Fix LoRa weight merging

Browse Source
This commit is contained in:
Kevin Kwok 2023-03-16 00:50:24 -07:00 committed by GitHub
parent b8c32be806
commit dde89950f3
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 6 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
export_state_dict_checkpoint.py
View File

@ -21,7 +21,12 @@ lora_model = PeftModel.from_pretrained(
torch_dtype=torch.float16,
)
lora_model.eval() # merge weights
# merge weights
for layer in lora_model.base_model.model.model.layers:
layer.self_attn.q_proj.merge_weights = True
layer.self_attn.v_proj.merge_weights = True
lora_model.train(False)
lora_model_sd = lora_model.state_dict()