ControlNet LoRA Brightness SDXL

Key Features:

• 🎨 Excellent brightness and pattern control across multiple scales (0.5-2.0)
• 🚀 196x smaller than full ControlNet: ~24MB vs ~4.7GB
• ⚡ Ultra-fast loading: LoRA weights load in <1 second
• 💡 Flexible scale control: Adjustable conditioning scale from 0.5 to 2.0+
• 🔄 Compatible with ControlLoRA v3: Uses the efficient ControlLoRA v3 architecture
• 📦 Minimal storage: All checkpoints + final model = ~490MB total
• 🖼️ Native SDXL resolution: Trained at 1024×1024
• 🎯 Production-scale training: 100,000 samples with PiSSA initialization

Intended Uses:

• Artistic QR code generation (scale 1.0-1.5 recommended)
• Image recoloring and colorization
• Lighting control in text-to-image generation
• Brightness-based pattern integration
• Watermark and subtle pattern embedding
• Photo enhancement and stylization

Training Details

Training Data

Trained on 100,000 samples from latentcat/grayscale_image_aesthetic_3M:

• High-quality aesthetic images
• Paired with grayscale/brightness versions
• Native resolution: 1024×1024 (SDXL native)

Training Configuration

Model Size Comparison

Usage

Installation

pip install diffusers transformers accelerate torch peft
# Install ControlLoRA v3
git clone https://github.com/HighCWu/control-lora-v3

Basic Usage

import torch
import sys
sys.path.insert(0, '/path/to/control-lora-v3')

from pipeline_sdxl import StableDiffusionXLControlLoraV3Pipeline
from model import UNet2DConditionModelEx
from diffusers import AutoencoderKL
from PIL import Image

# Load improved VAE (same as used in training)
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix",
    torch_dtype=torch.float16,
)

# Load UNet with LoRA support
unet = UNet2DConditionModelEx.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    subfolder="unet",
    torch_dtype=torch.bfloat16,
)
unet = unet.add_extra_conditions(["brightness"])

# Load SDXL Control LoRA pipeline with improved VAE
pipe = StableDiffusionXLControlLoraV3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    vae=vae,
    unet=unet,
    torch_dtype=torch.bfloat16,
)

# Load Control LoRA weights
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl", adapter_name="brightness")
pipe.to("cuda")

# Load grayscale/brightness control image
control_image = Image.open("path/to/grayscale_image.png")
control_image = control_image.resize((1024, 1024))

# Generate image
prompt = "a beautiful garden scene with colorful flowers and butterflies, highly detailed, professional photography, vibrant colors"

image = pipe(
    prompt=prompt,
    image=control_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    extra_condition_scale=1.0,  # Controls conditioning strength
    height=1024,
    width=1024,
).images[0]

image.save("output.png")

Adjusting Control Strength

The extra_condition_scale parameter controls how strongly the brightness map influences generation:

# Subtle control (scale 0.5-0.7)
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=0.5,
    ...
).images[0]

# Balanced control (scale 1.0-1.5) - Recommended for artistic QR codes
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=1.0,
    ...
).images[0]

# Strong control (scale 1.5-2.0)
image = pipe(
    prompt=prompt,
    image=control_image,
    extra_condition_scale=1.5,
    ...
).images[0]

Artistic QR Code Generation

import qrcode
from PIL import Image

# Generate QR code
qr = qrcode.QRCode(
    version=1,
    error_correction=qrcode.constants.ERROR_CORRECT_H,
    box_size=10,
    border=4
)
qr.add_data("https://your-url.com")
qr.make(fit=True)

qr_image = qr.make_image(fill_color="black", back_color="white")
qr_image = qr_image.resize((1024, 1024), Image.LANCZOS).convert("RGB")

# Generate artistic QR code (scale 1.0-1.5 works best)
image = pipe(
    prompt="a beautiful garden with colorful flowers and butterflies, highly detailed, professional photography",
    image=qr_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    extra_condition_scale=1.0,
    height=1024,
    width=1024,
).images[0]

image.save("artistic_qr.png")

Using Different Checkpoints

The model includes intermediate checkpoints from throughout training:

# Early checkpoint (25% - 25,000 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-781")

# Mid checkpoint (50% - 50,000 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-1562")

# Late checkpoint (75% - 75,000 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-2343")

# Near-final checkpoint (99% - 99,968 samples)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness",
                       subfolder="checkpoint-3124")

# Final model (100,000 samples, main branch - recommended)
pipe.load_lora_weights("Oysiyl/controlnet-lora-brightness-sdxl",
                       adapter_name="brightness")

Conditioning Scale Guide

The extra_condition_scale parameter controls how strongly the brightness map influences generation:

Recommended Scale Ranges

Performance Comparison

vs Full ControlNet (SDXL)

vs T2I Adapter (SDXL)

Checkpoint Progression Analysis

The model includes checkpoints from throughout training:

1. checkpoint-781: 25% complete (25,000 samples)
2. checkpoint-1562: 50% complete (50,000 samples)
3. checkpoint-2343: 75% complete (75,000 samples)
4. checkpoint-3124: 99% complete (99,968 samples)
5. Final model: 100% complete (100,000 samples - main branch)

Visual Comparison

Each comparison shows QR input + all 7 conditioning scales (0.25, 0.5, 0.7, 0.75, 1.0, 1.25, 1.5) for a specific checkpoint:

Checkpoint 781 (25% trained, 25,000 samples)

Checkpoint-781 Scale Progression

Checkpoint 1562 (50% trained, 50,000 samples)

Checkpoint-1562 Scale Progression

Checkpoint 2343 (75% trained, 75,000 samples)

Checkpoint-2343 Scale Progression

Checkpoint 3124 (99% trained, 99,968 samples)

Checkpoint-3124 Scale Progression

Final Model (100% trained, 100,000 samples) - Recommended

Final Model Scale Progression

Key Observations

All checkpoints show consistent, high-quality performance across scales. The progression analysis reveals:

1. Early Checkpoint (781 steps, 25k samples):

   • Strong pattern awareness from PiSSA initialization
   • Good balance between control and creativity
   • Recommended scales: 0.7-1.2

2. Mid Checkpoints (1562-2343 steps, 50k-75k samples):

   • Excellent balance between control and creativity
   • Stable pattern preservation across all scales
   • Recommended scales: 0.8-1.5

3. Final Model (3125 steps, 100k samples):

   • Maximum control capability with best generalization
   • Excellent pattern preservation at all scales
   • Recommended scales: 0.7-2.0
   • **Recommended for...