Manga109 Panel Balloon Text Yolov26 Segmentation

Model Description

This model is an Ultralytics-compatible YOLO26s instance segmentation model trained on Manga109-derived segmentation data. It predicts bounding boxes, class IDs, confidence scores, and pixel masks for manga page regions.

• Developed by: ShadowB / Abdelhadi Marjane
• Model type: Image segmentation / instance segmentation
• Architecture: YOLO26s segmentation model (yolo26s-seg.yaml)
• Base checkpoint: yolo26s-seg.pt
• Library: Ultralytics
• Task: Manga region instance segmentation
• Primary domain: Manga/comic page images
• Languages: Japanese manga pages. The model detects page regions visually; it does not read or translate text.
• License: MIT for this model repository. Dataset licenses and access rules may differ.
• Number of classes: 3
• Parameters: 11,436,269
• Stride: 8, 16, 32
• Checkpoint size: about 23.4 MB for best.pt
• Training date recorded in checkpoint: 2026-04-29
• Ultralytics version recorded in checkpoint: 8.4.43
• This model is part of github:sadowb/CuratorML a translation workspace that uses it for region detection.

Label Schema

The labels are stored in the model checkpoint and match the expected YOLO dataset names mapping:

Notes:

• Background is not an explicit class.
• text is the visual text region, not the OCR transcription.
• balloon is the container region around dialogue or narration text.
• frame is the panel/layout region, not necessarily a semantic scene label.
• Keep this class order unchanged in data.yaml, inference code, and downstream post-processing.

Recommended dataset config:

names:
  0: frame
  1: text
  2: balloon

Uses

Direct Use

Use this model to segment manga page regions from a page image. Direct outputs can be used to locate:

• panels/frames,
• text regions,
• speech or narration balloons.

Downstream Use

This model is designed to be one component in a larger manga translation or document-understanding system. A typical downstream flow is:

1. segment panels, balloons, and text regions,
2. associate text regions with balloons and panels,
3. run OCR on text regions,
4. reconstruct reading order,
5. translate text with surrounding visual/layout context,
6. inpaint or clean original text,
7. render translated text back into the page.

Example structured output expected by a downstream pipeline:

{
  "page": "example_page.jpg",
  "regions": [
    {
      "id": 1,
      "class_id": 0,
      "label": "frame",
      "confidence": 0.94,
      "bbox": [x1, y1, x2, y2],
      "mask": "..."
    },
    {
      "id": 2,
      "class_id": 2,
      "label": "balloon",
      "confidence": 0.91,
      "bbox": [x1, y1, x2, y2],
      "mask": "..."
    },
    {
      "id": 3,
      "class_id": 1,
      "label": "text",
      "confidence": 0.89,
      "bbox": [x1, y1, x2, y2],
      "mask": "..."
    }
  ]
}

Out-of-Scope Use

This model should not be treated as:

• an OCR model,
• a translation model,
• a reading-order model by itself,
• a general natural-image segmentation model,
• a legal/copyright analysis tool,
• a safety-critical segmentation system,
• a perfect layout parser for every comic style.

It only segments visible page regions. It does not understand text content, speaker identity, story context, or translation quality.

How to Get Started with the Model

Install dependencies:

pip install ultralytics pillow opencv-python

Run inference:

from ultralytics import YOLO

# Replace with your local path or the Hugging Face model ID after upload.
model = YOLO("best.pt")

results = model.predict(
    source="example_manga_page.jpg",
    imgsz=1280,
    conf=0.25,
    iou=0.7,
    retina_masks=True,
)

class_names = {
    0: "frame",
    1: "text",
    2: "balloon",
}

for result in results:
    if result.boxes is None:
        print("No regions detected.")
        continue

    for i, box in enumerate(result.boxes):
        class_id = int(box.cls[0])
        confidence = float(box.conf[0])
        bbox = box.xyxy[0].tolist()
        label = class_names.get(class_id, str(class_id))

        print({
            "index": i,
            "class_id": class_id,
            "label": label,
            "confidence": confidence,
            "bbox": bbox,
        })

    # Saves an annotated image with boxes/masks.
    result.save(filename="segmented_output.jpg")

For high-quality mask extraction in a manga translation pipeline, use retina_masks=True during inference so masks are returned at higher resolution.

Training Details

Training Data

This model uses a merged Manga109-derived segmentation dataset with three region classes: frame, text, and balloon.

Dataset Composition

The provided split audit records a book-level split across 109 manga groups:

The book-level split is important because random page-level splits can overestimate performance by leaking manga-specific style, art, and layout patterns between train and validation data.

Preprocessing

The training data was normalized into a YOLO-compatible segmentation layout with the following class mapping:

0: frame
1: text
2: balloon

Known preprocessing goals:

• merge Manga109-derived annotations into a common three-class schema,
• preserve separate panel/frame, text-region, and balloon masks,
• use a book-level split to better evaluate generalization across manga titles,
• train in an Ultralytics segmentation format compatible with yolo segment train.

Training Procedure

The model was trained on Kaggle with Ultralytics YOLO26s segmentation. The training script builds a book-level train/validation/test split, maps the labels to three classes (frame, text, balloon), and keeps overlap_mask=False because manga regions can sit inside each other.

Training used yolo26s-seg.pt as the starting checkpoint, image size 1280, batch size 8 across two GPUs, and MuSGD. The run completed 41 epochs and took 11h 13m overall. The checkpoint stores an Ultralytics training time value of 11.0054 hours, which reflects the active training budget rather than the full notebook runtime.

Training Hyperparameters

Model Size

Evaluation

Testing Data, Factors & Metrics

The available metrics are from the validation run recorded in best.pt, results.csv, and the validation artifacts in this repository.

• Validation split: book-level validation split
• Validation groups/books: 12
• Validation images: 1,468
• Test split available in audit: 14 groups/books, 1,488 images
• Metrics reported: box precision, box recall, box mAP, mask precision, mask recall, mask mAP
• Artifact source: /validationResultsOfMangaModel/results.csv and checkpoint train_metrics

Recommended factors for further evaluation:

• unseen manga titles/books,
• dense vs sparse pages,
• bordered vs borderless panels,
• large vs small balloons,
• small or dense text,
• heavy screentone regions,
• low-resolution or compressed pages,
• overlapping text/balloon/frame regions.

Results

The following values come from the best.pt checkpoint train_metrics field:

The final row in results.csv, epoch 41, records very similar overall metrics:

Per-Class Results

The local artifacts provided here include overall metrics, PR/F1/P/R curves, labels visualization, and confusion matrices. A per-class numeric mAP table was not present in results.csv.

To add per-class metrics, run a validation command that prints or exports per-class results, then update this table:

Suggested command when the dataset is available:

yolo segment val \
  model=best.pt \
  data=/path/to/data.yaml \
  imgsz=1280 \...