Illustrious LoRA Training Guide
Training Guide for PonyXL LoRA
Documenting my own process to create captions and train LoRAs, so a post that I can link to exists when needed~
Dataset Captions
"Explain my rather... unconventional, or even controversial, ethos on how to caption the dataset"
The majority of anime models on CivitAI were simply trained on datasets that only got auto-captioned, without any manual pruning at all, which results in the Trigger Words being a giant wall of text. So now, to recreate the look of a character alone, you already have to waste the entire default token count...
Instead, in my opinion, the caption should only consist of "variables" that do not define the subject. In essence, the model should learn to associate the trigger words with the subject. Take a character with blonde hair for example, the blonde hair
tag should be removed from the caption, as that "trait" should be linked to the character's name instead. Therefore, I prefer to manually prune the captions before training.
Some people would say this method reduces the flexibility of the model, as you can no longer control each individual detail precisely. However, this goes both ways. I have tried models where if you do not prompt for every single detail, the result ends up not resembling the character at all. So all in all, personally I'd rather use few tags to recreate a character.
Though recently, I found out that when training character pack LoRAs that contain multiple subjects, the model sometimes gets confused between the details, such as the hair colors and eye colors. This issue can most likely be fixed by using a higher Network Dimension, so that it can learn more information. But this also leads to a larger model size. Therefore, in the pursuit of keeping the storage size bearable, I decided that I will also include captions for eye color/hair color/hair style in a character pack LoRA.
The first model that adopts this method is the v3.0 of Assault Lily LoRA
Training Parameters
"Explain some of the parameters that worked the best for me after hours of testing"
mixed_precision: Use
bf16
if your GPU supports it for better resultsdynamo_backend: Improves training speed by ~5% if your GPU supports it
optimizer: Use
Prodigy
for automatically managedlearning rate
learning_rate: Set to
1.0
network_alpha: Set to
0.5
~0.25
ofnetwork_dim
d_coef: Set to
1.2
due to the need of higher learning rate caused bynetwork_alpha
lr_scheduler: Use
polynomial
to combatProdigy
's tendency of keeping learning rate highclip_skip: Use
2
for Ponydebiased_estimation_loss: This does seem to improve the color accuracy
---------------------------------------------------------------------------------------------------------------------
Type
|
Workflows
|
Stats |
622
0
|
Published |
2024-10-13 12:29:26 |
Base Model |
Pony
|
Usage Tips |
Clip Skip: 2
|
Trigger Words |
The file is a .json
config for Kohya_SS
Simply load the config and adjust the paths to your model and dataset
Also adjust the repeat/epoch count, so that each unique concept (character) gets trained for ~1500 steps in total
