ControlNet Anime Colorize

The goal of this experiment was to understand:

• How SDXL ControlNet learns colorization from line-based conditioning
• How different conditioning types (Canny vs Lineart) affect color consistency

Base Model

• Base model: cagliostrolab/animagine-xl-3.0
• Architecture: ControlNet SDXL
• Training framework: 🤗 Diffusers
• Precision: bf16

Conditioning Type

• Primary conditioning: Lineart / Canny-like edges
• Backgrounds are mostly white
• Line quality varies (mostly clean, some noisy samples)

Important limitation:
Lineart / Canny does not contain color information, which leads to unstable and drifting color predictions.

Dataset

• Size: ~14,000 image pairs
• Format:
  • Original image (color)
  • Conditioning image (lineart / canny)
  • Prompt (caption)

Known dataset issues

• Some lineart images are noisy or inconsistent
• Images are resized to square resolution (possible cropping artifacts)
• No explicit color supervision
• No palette or region-level color constraints

Training Configuration

Typical training setup:

resolution: 768
train_batch_size: 2
gradient_accumulation_steps: 2
effective_batch_size: 4
learning_rate: 2e-5
lr_scheduler: cosine
max_train_steps: 6000–8000
mixed_precision: bf16

--- license: mit datasets: - SubMaroon/danbooru-lineart base_model: - cagliostrolab/animagine-xl-3.0 --- # Experimental ControlNet (Low Quality / Research Prototype) > **Experimental model. Low quality. Not intended for production use.** > This ControlNet was trained as a research experiment to explore line-based conditioning and colorization behavior in SDXL anime models. --- ## Model Summary This repository contains an **experimental ControlNet for SDXL**, trained on anime-style images. The model is **not stable**, shows **inconsistent color behavior**, and should be treated as a **research prototype** rather than a finished or polished solution. The goal of this experiment was to understand: - How SDXL ControlNet learns **colorization from line-based conditioning** - How different conditioning types (Canny vs Lineart) affect **color consistency** --- ## Base Model - **Base model:** `cagliostrolab/animagine-xl-3.0` - **Architecture:** ControlNet SDXL - **Training framework:** 🤗 Diffusers - **Precision:** `bf16` --- ## Conditioning Type - Primary conditioning: **Lineart / Canny-like edges** - Backgrounds are mostly white - Line quality varies (mostly clean, some noisy samples) > Important limitation: > Lineart / Canny **does not contain color information**, which leads to unstable and drifting color predictions. --- ## Dataset - Size: ~**14,000 image pairs** - Format: - Original image (color) - Conditioning image (lineart / canny) - Prompt (caption) ### Known dataset issues - Some lineart images are **noisy or inconsistent** - Images are resized to square resolution (possible cropping artifacts) - No explicit color supervision - No palette or region-level color constraints --- ## Training Configuration Typical training setup: ```bash resolution: 768 train_batch_size: 2 gradient_accumulation_steps: 2 effective_batch_size: 4 learning_rate: 2e-5 lr_scheduler: cosine max_train_steps: 6000–8000 mixed_precision: bf16