comfyui-tag-rag

comfyui-tag-rag

ComfyUI custom nodes for tag-based prompt RAG using CSV tag datasets, lexical/optional FAISS retrieval, and local llama.cpp generation.

Status

Experimental / Pre-Alpha. The current usable path is:

CSV tags -> TagRAG Index -> lexical/semantic retrieval -> TagRAG Prompt/Chat/Image -> native llama-cpp-python generation by default or alternate llama-server generation

Implemented now:

• CSV loading with encoding fallback.
• Normalized TagRecord / TagCatalog data model.
• Lexical retrieval over tags, aliases, categories, descriptions, exclusions, and weights.
• Basic TagRAG Index node that returns a serialized JSON index_ref.
• Basic TagRAG Prompt node that reloads the CSV from index_ref, retrieves tags, builds a deterministic one-shot prompt, and calls native llama-cpp-python by default or a local llama-server endpoint when selected.
• Native llama-cpp-python embeddings for semantic/hybrid retrieval when an embedding-capable GGUF is provided, plus alternate llama-server embeddings through /v1/embeddings or legacy /embedding.
• Optional FAISS semantic indexes persisted beside the CSV under .tag_rag_indexes/<index_name>/.
• Semantic and hybrid retrieval modes when faiss-cpu and an embedding endpoint are available.
• Basic TagRAG Chat node with bounded in-memory conversation history and RAG retrieval per turn.
• TagRAG Image node that accepts existing tag strings as the primary path, can optionally run built-in WD14-style ONNX tagging when image extras/model files are manually installed, retrieves related CSV tags, and optionally refines through llama-server.
• Native llama-cpp-python in-process GGUF generation backend for Prompt, Chat, and Image refinement is the normal/default path; llama-server remains supported.

Future:

• Advanced vision-language image conversation/collaboration workflows.

Goal

Help users craft better ComfyUI image-generation prompts by combining:

• A personal CSV tag dataset.
• Retrieval over that dataset.
• Local llama.cpp-based text generation.
• Optional chat and tag-to-prompt/image-to-tag workflows.

Quick Start: Lexical Retrieval + Prompt Generation

1. Install the package into your ComfyUI custom_nodes/ directory.

2. Install core requirements:

   pip install -r requirements.txt
   

3. Provide an existing local generation GGUF through model_path for the default native backend. No model is downloaded automatically.

   Or start/point to an existing local llama.cpp server if you prefer generation_backend=llama_server:

   http://127.0.0.1:8080/v1/chat/completions
   

4. Add TagRAG Index in ComfyUI.

   • Set csv_path to your CSV file.
   • Set tag_column to the column containing tag text.
   • Optional columns: aliases, category, description, source, weight.
   • The node returns a JSON index_ref string.

5. Wire index_ref into TagRAG Prompt.

6. Enter an idea, model_path, top_k, max_tokens, and temperature.

7. Leave generation_backend=llama_cpp_python for the default path, or switch to llama_server and use base_url/model.

8. Use the positive_prompt output in your ComfyUI generation workflow.

No FAISS or embedding model/server is required for the lexical retrieval path.

Optional: Semantic or Hybrid Retrieval

1. Normal ComfyUI installs use requirements.txt, which now includes FAISS and native/image dependencies. If you maintain split installs, semantic retrieval needs:

   pip install -r requirements-rag.txt
   

2. Choose an embedding backend in TagRAG Index.

   Native default for semantic/hybrid:

   embedding_backend: llama_cpp_python
   embedding_model_path: path to an embedding-capable GGUF
   

   Alternate llama-server backend:

   OpenAI-compatible: http://127.0.0.1:8080/v1/embeddings
   Legacy llama.cpp:  http://127.0.0.1:8080/embedding
   

3. In TagRAG Index, set retrieval_mode to semantic or `hy