rafacost-comfy

ComfyUI-DreamOmni2-GGUF

A ComfyUI custom node for DreamOmni2 GGUF multimodal models — powered directly by llama-cpp-python, no external executables required.

⚠️ Work in Progress — use at your own risk. (Or better: fork and help improve it.)

---

✨ Features

• Run DreamOmni2 GGUF models natively inside ComfyUI.

• Full image + text multimodal support through the llama-cpp-python backend.

• Accepts up to four image inputs simultaneously.

• Outputs either:

  • conditioning embeddings (for generation workflows), or
  • text descriptions (for captioning or analysis).

• Built-in seeded cache system for deterministic results across sessions.

• No dependency on external binaries or CLI tools.

---

🧩 Prerequisites

Requires Python ≥ 3.12 and a working ComfyUI ≥ 0.3.66.

Install dependencies (CPU or CUDA builds supported):

pip install -r requirements.txt

requirements.txt

torch>=2.2.0
numpy
Pillow
scikit-build-core
llama-cpp-python>=0.3.16

For CUDA acceleration, install the matching wheel from llama-cpp-python (CUDA builds).

Example (Windows CUDA 12.x):

pip install llama_cpp_python-0.3.16+cu124-cp312-cp312-win_amd64.whl

---

🧠 Installation

1. Clone this repository into your ComfyUI custom_nodes directory:

   git clone https://github.com/rafacost/rafacost-comfy.git
   

2. Download the DreamOmni2 GGUF and mmproj models from Hugging Face.

3. Place them in:

   ComfyUI/models/unet/
   ├── DreamOmni2-Vlm-Model-7.6B-Q5_K.gguf
   ├── DreamOmni2-Vlm-Model-7.6B-f16-mmproj.gguf
   ├── flux1-kontext-dev-Q5_K.gguf
   ComfyUI/models/loras/
   ├── DreamOmni2-7.6B-Edit-Lora.safetensors
   ├── DreamOmni2-7.6B-Gen-Lora.safetensors
   

---

⚙️ Usage

1. Launch ComfyUI.

2. Add the DreamOmni2-VLM node under rafacostComfy / VLM.

3. Configure the node:

   • Model – select your DreamOmni2 .gguf model.
   • MMProj Path – select your vision projection .gguf.
   • Images – connect up to four Load Image nodes.
   • Prompt – enter a description or instruction.
   • Seed – for deterministic results (cache linked).
   • Use Cache – toggle to reuse previous generations.
   • As Conditioning – output embeddings instead of raw text.

4. Optionally connect a DreamOmni2 Output Node to visualize the generated text inside the workflow.

---

🧪 Example

Use the sample workflow provided in workflows/ or connect images manually. Outputs appear both in the ComfyUI graph and in the terminal console.

---

⚙️ Troubleshooting

Issue	Cause	Fix
ImportError: cannot import Qwen25VLChatHandler	Missing or outdated llama-cpp-python	pip install --upgrade llama-cpp-python>=0.3.16
Model loads but crashes	Out-of-memory	Try lower quant (e.g. Q4_K)
No output text	Prompt too short / token limit	Increase max_tokens
Different outputs for same seed	Cache disabled or model reset	Enable Use Cache
“NoneType object” errors	Missing mmproj	Verify both GGUF files are present
Low Image Quality	Low Base Model GGUF quant	Try a higher GGUF quant for BaseModel (eg. use flux_kontext:Q8_0)
Low Prompt Adherence