FLUX.1 LoRA training (20250126)

• An Nvidia GPU with at least 24GB of Video RAM

• Using Flux.1-Dev, requires acceptance of the terms of use.

• A HuggingFace account to download the weights.

Preliminary steps

Code retrieval and virtualenv creation

mkdir ~/Flux_Lora
cd ~/Flux_Lora

# Obtain the source and the code's submodules
git clone https://github.com/ostris/ai-toolkit.git
cd ai-toolkit
git submodule update --init --recursive

# Create and enable a python virtual environment to place all the needed packages
python3.12 -m venv venv
source venv/bin/activate
# for future use, re-enabling the venv can be done by running the source command again

# Confirm python3 and pip3 are from our venv
which python3
which pip3

# Install the required packages
pip3 install torch
pip3 install -r requirements.txt
pip3 install mediapipe peft 
# mediapipe appears to be needed for Dev, while peft will be needed for Schnell

Hugging Face token

# Install the CLI using brew
brew install huggingface-cli

# Confirm the token is valid and add it to the default HF_HOME at ~/.cache/huggingface
huggingface-cli login
# Answer no to "Add token as git credential" as this is a "download-only" token
# This will store the token in ~/.cache/huggingface/token
# All models retrieved from HF's hub will end up in ~/.cache/huggingface/hub

Dataset preparation

mkdir ~/Flux_Lora/training_images-tok

• Have at least 12 face photographs

• Each photograph should have altering conditions so the model training can differentiate the person from the background or its clothing. As such, use shots with altering angles of the face, lighting conditions, background, clothes, and ornaments (limit the ones hiding your hair, but if wearing glass, take images with and without those).

• The script requests that you name each file image[number].[extension] increasing number as needed: image1.jpg, image2.png … image25.jpg

• Each image file must have a corresponding .txt file with a description of the image. Because the developer allowed for text replacement using the [trigger] keyword, each file can contain the same value. For example, ours contains a photo of a [trigger] man. The content can be extended to be more descriptive, but this is sufficient to train the LoRA.
• Each image must have a matching .txt file: image1.jpg with image1.txt … image25.jpg and image25.txt

LoRA Training

Flux.1-Dev

cd  ~/Flux_Lora/ai-toolkit
# create an output directory for the trained LoRA(s) 
mkdir ../trained_LoRA
# copy the example training yaml and rename it
cp config/examples/train_lora_flux_24gb.yaml config/flux_dev-tok.yaml

• In config, we modified the name: "flux_dev-tok" to reflect that we are creating a Flux.1-Dev LoRA for a tok prompt trigger word.

• We are modifying training_folder: "../trained_LoRA" to use our created directory. The script will create a flux_dev-tok folder (it uses the value of name) to store the training and samples.

• Uncomment and set the trigger_word to a value you will use in your prompts. p3r5on or tok are commonly used values. The image generation text prompts will recognize this value. Using different values for different persons (it is recommended to always use a single non-existing word). Here we will use trigger_word: "tok” . This trigger word will be replaced inside the imageNUM.txt automatically and in the sample prompts section.

• The save section specifies how often the training saves .safetensors checkpoints. Each is usable, so if you decide a specific checkpoint gives better results (based on the samples generated), you can use that file instead of the final one. It is recommended that the sample_every generation match the save_every value.

• In the datasets section we adapt the folder_path to where our training images are located: folder_path: "../training_images-tok"

• In the train section, we increased the number of steps to 4000.
• The tool will generate checkpoints at save_every steps. If the samples deteriorate as training steps increase, we can select an earlier checkpoint file instead of the later one(s).

• We are not altering the model section.
• If already present in the HF cache, the file will not be re-downloaded.
• You must have agreed to the terms for the FLUX.1-Dev model for it to be accessible.

• In the sample section, we keep the sample_every to match the save_every value. We can comment out (#) some prompts while adding [trigger] to some. The idea is to see how good the model is at training the LoRA on the images we provided, not to demonstrate the capabilities of FLUX.1-dev. For example:

        prompts:
          # you can add [trigger] to the prompts here and it will be replaced with the trigger word
          - "a woman holding a sign that says, 'this is a sign'"
          - "a [trigger] man holding a sign that says 'AI is fun'"
          - "a [trigger] man as a Jedi warrior, deep blue lightsaber, dark purple background with some multicolor neon light reflecting"
          - "a [trigger] man as a green lantern, cosmic background, creating a dragon from his ring, majestic, rule of third"
          - "a [trigger] man as Conan the barbarian, snow-covered fields, sword in hand, defiant, dynamic pose, golden ratio, asymmetric composition, photorealism, cinematic realism"
          - "a portrait of a [trigger] man as Superman flying in space, yellow sun in the background, majestic, comic style"
          - "a [trigger] man as a cyberpunk warrior, anime style"
          - "professional photograph portrait of a [trigger] man, black and white"
          - "a [trigger] man wearing sunglasses riding a high-speed motorbike, hyper-maximalist, octane render"
          - "a [trigger] man, lucid dream-like 3d model, game asset, blender, unreal engine, rule of thirds, wide angle shot, looking off in distance style, glowing background, vivid neon wonderland, particles, blue, green, orange"

Here, only our first prompt will be consistent from sample generation to sample generation (the seed is set and not random). All the other generations will show us the model's improvements at generating a [trigger] man. We have a total of 10 prompts.

# Make sure you are running the create virtual env
# if needed: cd ~/Flux_Lora/ai-toolkit; source venv/bin/activate
time python3 ./run.py config/flux_dev-tok.yaml

Flux.1-Schnell

cd  ~/Flux_Lora/ai-toolkit
# create an output directory for the trained LoRA(s) if it does not already exist
mkdir ../trained_LoRA
cp config/examples/train_lora_flux_schnell_24gb.yaml config/flux_schnell-tok.yaml

• training_folder: "../trained_LoRA"

• trigger_word: "tok”

• folder_path: "../training_images-tok"

• steps: 4000

# Make sure you are using the created virtualenv
# if needed: ~/Flux_Lora/ai-toolkit; source venv/bin/activate

# Run the training
time python3 ./run.py config/flux_schnell-tok.yaml

Using the trained models: ComfyUI

With ComfyUI

Flux.1-Dev LoRA with ComfyUI

Flux.1-Schnell LoRA with ComfyUI

Extended workflow

“Atomix Flux UNet” modified Workflow

Useful Workflows

• zyd232's Workflow for Flux - ControlNet Upscale + Expression Editor + Face Detailer + Hand Fix

• Dickson's Sci-Fi Enhance Upscale

Revision History

• 20250126-0: Updated with January run + added workflows

• 20240824-0: Added more complex workflow, integrating ComfyUI Manager.

• 20240818-0: Initial release