AI/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2024-10-01 01:26:03 -04:00
Code Issues Packages Projects Releases Wiki Activity

LoRA: Fix error "Attempting to unscale FP16 gradients" when training (#5268)

Browse Source
This commit is contained in:
ilya sheprut 2024-01-17 23:11:49 +03:00 committed by GitHub
parent 535ea9928a
commit 4d14eb8b82
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
modules/training.py
View File

@ -605,6 +605,11 @@ def do_train(lora_name: str, always_override: bool, q_proj_en: bool, v_proj_en:
control.should_training_stop = True
print(f"\033[1;31;1mStop Loss {stop_at_loss} reached.\033[0;37;0m")
# Fix training for mixed precision models
for param in shared.model.parameters():
if param.requires_grad:
param.data = param.data.float()
trainer = transformers.Trainer(
model=lora_model,
train_dataset=train_data,