creative-code-comfyui

Algorithmic Attention (XAI) Tool Documentation

flower

My tool uses pre-trained image classifiers using gradient-based saliency methods from Google's PAIR saliency library, exposing the hidden attentional logic of neural networks as visual artifacts. Built as a custom ComfyUI node set.

I've been interested in Explainable AI (XAI) for some time now, and wanted to actually UNDERSTAND how it works.

XAI:

a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms (IBM, 2022)

I do research with phD student Katelyn Morrison, and learned about her work with saliency mapping, which is a visual way to show what areas of an image are most significant for ML classification.

map

When I saw these maps, I immediately wondered if I could make art using ML classifications.

So, I built this tool.

The core question my tool seeks to answer: when a model classifies an image, which pixels drove that decision? (and, how can we make art with this algorithmic attention).

---

Table of Contents

1. How It Works
2. Models
3. Methods
4. Composition Modes
5. Colormaps
6. Other Parameters

---

How It Works

node

All seven supported models are convolutional neural networks (CNNs) trained on ImageNet, which is a dataset of 1.2 million images across 1,000 categories. The training process adjusts millions of numerical weights until the network can reliably distinguish between categories like "labrador retriever," "coffee," and "duck."

What makes them useful for saliency visualization is that they're all differentiable, built from mathematical operations (matrix multiplications, convolutions, activation functions), so you can calculate the derivative of the classification output with respect to any input pixel. That derivative is the gradient: if I changed this pixel slightly, how much would the model's confidence change? High gradient = the model is paying attention to that pixel.

The models differ in architecture, which changes the character of their attention; where they look, how diffuse or concentrated it is, how coarse or fine-grained...

---

Models

The model is the "brain" of the node.

Demo settings: method: Guided IG · composition: pure_white · colormap: magma · intensity: 0.75 · contrast: 1.5 · threshold: 0.25 · class_index: -1 · input_size: 224 · use_smoothgrad: true · smoothgrad_samples: 25

TLDR: Coarse and blunt (VGG) → distributed and semantic (ResNet) → diffuse and layered (DenseNet) → multi-scale (Inception) → sparse and shortcut-prone (MobileNet/EfficientNet)

---

VGG16 and VGG19

Among the oldest and simplest networks still in common use, the design is a deliberately minimal stack of small 3×3 convolutional filters in sequence, going deeper and deeper, with max-pooling layers periodically halving the spatial resolution. VGG16 has 16 layers with learned weights; VGG19 has 19. Both end with three large fully-connected layers that make the final classification decision.

Each convolutional layer detects increasingly abstract features. The early layers find edges and color gradients, middle layers find textures and simple shapes, and later layers find object parts. The fully-connected layers at the end combine everything into a single classification verdict. Because the architecture is purely sequential with no shortcuts, everything must compress through a bottleneck.

VGG models have the largest, most spatially coarse attention patterns of any model here. Because the architecture is old and inefficient, it has to "look hard" at large regions rather than pin