Anima-Control-Pose

There’s also a little ComfyUI node that gives you one more thing to try when a pose almost works but not quite. Usually you’d just keep rerolling the seed. This lets you also change how the skeleton itself is drawn: thicker lines, a colored puppet version, a heatmap, or with the hands or face dropped. The plain thin one is still what you want most of the time, since that’s the only kind it actually trained on, but every so often a different render gets a stubborn pose to land when the seed won’t. It pulls the pose off a reference photo for you too, so you’re not drawing skeletons by hand.

For the technically curious: it’s still a rank-16 control-LoRA sitting on the frozen Anima model with a zero-init skeleton embedder, so at strength 0 you get the base model back exactly, untouched. The real changes are the bigger corpus (about double Preview-1’s ~3,900 examples) and training across all three resolutions for 8 epochs instead of 512-only. On held-out poses the pose-accuracy score went up at every size, with the biggest gains at 768 and 1024.

What’s still off, honestly

It doesn’t always follow the skeleton. Even a clean, correct one. I’ve had a clear walking pose come out as someone just standing there.

Part of why: a thin stick figure on a black background isn’t really how anime gets drawn, so the model only half-reads it. And the ugly stuff (fused hands, general mush) shows up right where the skeleton is busiest.

Anything dynamic — running, jumping, sitting — is still the shakiest.

On short or lazy prompts the style drifts toward one samey default. Give it a proper prompt and it comes back.

What I’m working on right now

Preview-3 is about that “doesn’t follow the skeleton” problem, which turned out to be more interesting than just “train it harder.” The thin-lines-on-black signal seems kind of foreign to the model, so before I sink a couple days into a real training run, I ran a small bake-off on the pose signal itself: take the same poses, draw them a few different ways — the current thin lines, a thicker high-contrast one, a colored body-part “puppet,” and a depth/mannequin look — then measure which one Anima actually follows, and how much quality it costs. That’s how I’ll decide what the skeleton should even look like. The smaller fixes ride along with it: a lower default strength, dropping the noisy hand points, and more varied captions so the look stops flattening.

The other half is the detector. The one I’m using was made for photos, not anime, so on art it hands me messy skeletons, and a messy skeleton wrecks everything after it. So Preview-3 is probably going to wait until I build a pose detector meant for anime, a sort of “DWPose for anime,” which I’m starting on right after this. Preview-3 is then the actual retrain: clean detector, whichever render won the bake-off, and a lot more dynamic poses in the data. No date on it. It hangs on that detector working.

Then 1.0

If Preview-3 lands clean, that’s the one that drops the “preview” label, and it’ll train on a good deal more data again, not just the same pile pushed further.

Also available on Hugging Face: https://huggingface.co/Claquasse/Anima-Control-Pose

⚠️ The main file here is the adapter weights (adapter_model.safetensors). The ComfyUI custom node, both workflows, and the skeleton script are attached as anima_control_tools.zip , with pose_control_demo.json also attached on its own for drag-and-drop. You have an INSTALL.txt file inside of the archive to guide you. Everything is mirrored in the Hugging Face repo above.

Anima Control — Pose (Preview-1)

⚠️ Preview-1 — experimental. This is an early, lightly-trained release of pose control for Anima v1.0. Because it is a preview, it will often generate bad images: deformed or split bodies, wrong or extra limbs, and similar artifacts. It simply has not had enough training and refining yet. Treat it as a proof of concept, not a finished tool. It is not production-ready for now. Non-commercial use only (inherits the Anima base model license). Behavior and weights may change.

A native pose control adapter for the Anima v1.0 image model: condition generation on a skeleton pose map so the subject follows a target pose.

Abstract

Anima Control adds spatial conditioning to the frozen Anima v1.0 diffusion transformer without retraining the base model. Preview-1 ships the first component, pose, as a channel-concat control-LoRA. A small, zero-initialized embedder injects a VAE-encoded skeleton at the model input, and a low-rank adapter on the transformer blocks lets the frozen network act on that signal. Because the embedder starts at zero, an untrained or zero-strength adapter is an exact no-op: the output is identical to the base model. Control is optional and off by default.

Highlights

Pose conditioning for Anima v1.0: drive the subject's pose from a skeleton map.

Zero-init no-op: at strength = 0 the output is exactly the base model.

Small trainable footprint: only the control embedder and a rank-16 adapter train; the base stays frozen.

Part of a shared control platform: the same harness is meant to host other control types (see roadmap below).

Method

The adapter has one fusion point at the model input and trains two small parts.

Conditioning. The base VAE encodes the skeleton pose map into a control latent, the same latent space as the noisy image, so the control stays spatially aligned with the generation.

Fusion ( ControlEmbedder + ControlInitialLayer ). A zero-initialized ControlEmbedder (a patch projection on the control latent) produces control tokens that are added to the frozen base patch-embed output inside an overridden ControlInitialLayer . This is the only place the forward pass changes. Zero-initialization means training starts as an exact no-op (output == base) and the control contribution grows only as it earns loss.

Trainable parameters. The ControlEmbedder plus a rank-16 low-rank adapter on the transformer Block s. The base transformer, text encoder, and VAE are frozen.

Objective. The base model's native flow-matching objective, unchanged.

skeleton ─▶ VAE ─▶ control latent ─┐ ▼ (+ zero-init ControlEmbedder) noisy latent ─▶ patch-embed ─▶ [ControlInitialLayer] ─▶ Block×N (+ rank-16 LoRA) ─▶ output

Training

Data. 3,914 (image, skeleton, caption) triples. Images are generated by Anima from a broad prompt distribution; skeletons are rendered from each image's detected keypoints (DWPose, COCO-WholeBody, black background); captions are quality-prefixed tag lists.

Configuration.

SettingValueResolution512, aspect-ratio bucketed (AR 0.5–2.0, 7 buckets)Adapter rank16Learning rate1e-4Epochs10Control dropout0.1Precisionbf16Optimizeradamw (optimi)

Final training loss: ≈ 0.13 (denoising MSE, mean over the final 400 optimizer steps); stable across all 10 epochs.

Results

Preliminary — measured on 10 held-out full-body poses (fresh generations not seen in training). Pose agreement is scored by re-detecting keypoints on each generated output and comparing to the target skeleton (PCK@0.1 = fraction of keypoints within 10% of the pose bounding-box diagonal).

Each row: the input reference, the DWPose skeleton extracted from it, base Anima at the same prompt and seed, then the pose-controlled output. Base ignores the skeleton; control follows it.

Control roughly triples body-pose agreement (0.20 → 0.59); several held-out poses (standing, waving, hands-on-hips) reach near-perfect adherence.

With control off (strength 0.0) the output matches base Anima whatever the skeleton: the adapter is a clean no-op when disabled.

Strength sweep (0.0 / 0.5 / 1.0): raising strength moves the subject from the base pose toward the target skeleton; 0.5 is a partial blend, 1.0 follows the skeleton.

Limitations: adherence is strongest on upright / static poses; highly dynamic poses (running, jumping) are followed only partially at this preview's scale and resolution.

Usage (ComfyUI)

Preview-1 runs as a small ComfyUI custom node, Anima Control Apply ( AnimaControlApply , category AnimaControl ).

Install

Download the adapter ( adapter_model.safetensors , the file attached to this CivitAI model) into ComfyUI/models/loras/ .

From the Hugging Face repo , copy the comfyui/anima_control_lora/ folder into ComfyUI/custom_nodes/ , then restart ComfyUI. That folder is the whole node: init.py and its one helper, control_embedder.py .

Load pose_control.json (also in the HF repo) from the ComfyUI workflow menu, or build the graph by hand (below).

One file does two jobs. adapter_model.safetensors holds both a low-rank adapter ( lora.* keys) and the control embedder ( control_embedder.* keys). LoraLoaderModelOnly reads the first set, Anima Control Apply reads the second. A bare filename in control_embedder_path is resolved against models/loras/ , so both nodes just point at adapter_model.safetensors .

Two workflows are in the HF repo:

pose_control_demo.json — the easiest way to test everything. Drop in any reference image; it runs DWPose to extract the skeleton, then generates the same prompt twice, once on base Anima and once with pose control, so you can compare side by side. The DWPose step needs the comfyui_controlnet_aux nodes.

pose_control.json — the core control graph, where you supply a skeleton image yourself.

Prepare a skeleton

The control input is a skeleton pose map: DWPose keypoints (COCO-WholeBody) drawn on a black background. You can make one two ways:

A DWPose preprocessor node (for example, from comfyui_controlnet_aux ) run on a reference image inside ComfyUI.

The bundled scripts/render_skeletons.py (in the HF repo), which needs pip install rtmlib opencv-python :

python scripts/render_skeletons.py --imgs --out

Graph

Load base Anima with your usual model loader.

LoraLoaderModelOnly : set lora_name to adapter_model.safetensors .

VAEEncode : encode the skeleton image to a LATENT (this is the control_latent ).

Anima Control Apply: inputs are model , control_latent , control_embedder_path ( adapter_model.safetensors ), and strength . It returns a patched MODEL .

Sample from the patched model as usual.

Strength. 0.0 is the base model with no control; 1.0 follows the skeleton. The range is 0 to 2. Higher values track the pose more closely but can cost some image quality.

Limitations

Preview quality. Lightly trained; pose adherence may be inconsistent and is still under evaluation.

512×512 only. The adapter is trained at 512, and the control latent has to line up spatially with the generation latent, so the skeleton input and the output are both fixed at 512×512. Other resolutions are not supported yet: off-512 the latents do not match and the pose is not tracked. Fine detail (hands, faces) is also limited at this resolution.

Single input injection. Control is asserted once, at the input; at low strength the model may under-follow the pose. Raising strength helps but can reduce image quality.

Pose only. Other control types are roadmap, not in this release.

Non-commercial. Inherits the Anima base model license.

The platform & its v1 components

Pose is the first…

(This model is for Anima-V1.0, the "preview2" is the version of this model not Anima) ⚠️ To use this model, you will need custom nodes and workflow. Find them inside "anima_control_tools.zip". Having an issue? Ask in comments (and they may already cover…