SD14 Pathology ControlNet

--- base_model: - CompVis/stable-diffusion-v1-4 - RiddleHe/SD14_pathology_lora library_name: diffusers license: creativeml-openrail-m inference: true tags: - stable-diffusion - stable-diffusion-diffusers - text-to-image - diffusers - controlnet - diffusers-training --- <!-- This model card has been generated automatically according to the information the training script had access to. You should probably proofread and complete it, then remove this comment. --> # controlnet-RiddleHe/SD14_pathology_controlnet These are controlnet weights trained on CompVis/stable-diffusion-v1-4 with new type of conditioning. You can find some example images below. prompt: A histopathology image of breast cancer tissue.  prompt: An image of breast cancer histopathology with detailed cellular structures.  ## Intended uses & limitations #### How to use ```python controlnet = ControlNetModel.from_pretrained("RiddleHe/SD14_pathology_controlnet", torch_dtype=torch.float16) pipe = StableDiffusionControlNetPipeline.from_pretrained( "RiddleHe/SD14_pathology_base", controlnet=controlnet, torch_dtype=torch.float16 ) pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config) pipe.to('cuda') prompt = "A histopathology image of breast cancer tissue." mask = mask.convert("RGB") # Provide a mask generator = torch.Generator(device='cuda').manual_seed(42) with torch.no_grad(): out = pipe(prompt, image=mask, num_inference_steps=70, num_images_per_prompt=3, generator=generator).images ``` #### Limitations and bias [TODO: provide examples of latent issues and potential remediations] ## Training details The model is trained on 28216 image-mask pairs from the BRCA breast cancer dataset. Input is mask and output is image. Mask is a single channel image with integer values from 0 to 21 representing 22 classes, eg. 1 representing tumor, 2 representing stroma.