You can deploy this model and start earning money today!

RichardScottOZ/
comic-strip-encoder-v1

narrative-understanding

2503.08561

2403.03719

mit

Views

No views yet

Comic Strip Encoder v1 (Stage 4)

This model is a Transformer sequence encoder designed to generate narrative-aware, contextualized embeddings of comic book page strips. It serves as "Stage 4" of the Comic Analysis Framework v2.0.

Where comic-panel-vlm-v1 (Stage 3) generates a 512-dimensional embedding per panel in isolation, this model takes a full page's worth of panel embeddings (a strip) and runs them through a Transformer encoder. Every output embedding is then conditioned on the panels surrounding it — the model has learned what a panel means in the context of the story around it. The outputs are:

contextualized_panels (N, 512) — per-panel embeddings enriched with sequential narrative context
strip_embeddings (512,) — a single vector summarising an entire page/strip

These are intended as the primary inputs for downstream retrieval, reranking, and narrative analysis tasks (Stage 5).

Model Architecture

The comic-strip-encoder-v1 is a BERT-style Transformer Encoder (Stage4SequenceModel):

Input Projection: Linear layer mapping 512-d panel embeddings into the model's d_model space.
Positional Encoding: Learned positional encodings for panel sequence order.
Panel Sequence Transformer:
- 6 Transformer encoder layers
- 8 attention heads
- Pre-norm (LayerNorm before attention) for training stability
- Attention masking for variable-length strips (max 16 panels)
Strip Aggregation: A learned [CLS]-style query attends over all panel outputs to produce a single strip-level vector.
Task-Specific Heads (7 total, used during training):

Head	Task	Paper
`ReadingOrderHead`	Pairwise panel ordering (adjacency matrix)	ComicsPAP
`PanelPickingHead`	Select missing panel from candidates	ComicsPAP
`CharacterCoherenceHead`	Visual identity consistency across panels	ComicsPAP
`VisualClosureHead`	Action continuation plausibility	ComicsPAP
`TextClosureHead`	Dialogue continuation plausibility	ComicsPAP
`CaptionRelevanceHead`	Text-visual alignment scoring	ComicsPAP
`TextClozeHead`	Select correct dialogue given visual context	Text-Cloze

At inference time only the Transformer backbone + strip aggregator are required for embedding generation. The task heads can be used directly for scoring tasks.

Training Data & Methodology

The model was trained on sequences of panel embeddings generated by comic-panel-vlm-v1 across approximately 1 million comic pages, filtered for narrative/story content by Stage 2 (CoSMo PSS).

Research Foundation

ComicsPAP (arXiv:2503.08561): Five discriminative tasks for sequential comic understanding. State-of-the-art LMMs perform near chance on these tasks; domain-trained sequence models are necessary.
Text-Cloze (arXiv:2403.03719): Multimodal transformers outperform RNNs by ~10% on dialogue cloze tasks; domain-adapted encoders are critical.

Training Objectives

L_total = Σ(weighted task losses) + 0.5 * L_contrastive + 0.3 * L_reading_order

Task weights during multi-task training:

python

task_weights = {
    'panel_picking':        1.0,   # Primary ComicsPAP task
    'text_cloze':           1.0,   # Primary Text-Cloze task
    'reading_order':        0.7,
    'visual_closure':       0.8,
    'text_closure':         0.8,
    'character_coherence':  0.5,
    'caption_relevance':    0.5,
}

Key design choice — discriminative not generative: candidates are selected from a pool rather than generated, following the ComicsPAP framework. This makes training tractable and evaluation unambiguous.

Usage

The codebase is available at the Comic Analysis GitHub Repository under src/version2/stage4_sequence_modeling_framework.py.

Example: Generating Strip & Panel Embeddings

python

import torch
from stage4_sequence_modeling_framework import Stage4SequenceModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# 1. Initialize model
model = Stage4SequenceModel(d_model=512, num_layers=6, nhead=8).to(device)

# Load weights from Hugging Face
state_dict = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/RichardScottOZ/comic-strip-encoder-v1/resolve/main/best_model.pt",
    map_location=device
)
model.load_state_dict(state_dict['model_state_dict'])
model.eval()

# 2. Inputs: panel embeddings from comic-panel-vlm-v1
# panel_embeddings: (B, N, 512)  — N panels on one page, up to 16
# panel_mask:       (B, N)       — True where panel exists

panel_embeddings = torch.randn(1, 6, 512).to(device)  # 1 page, 6 panels
panel_mask = torch.ones(1, 6, dtype=torch.bool).to(device)

# 3. Generate embeddings
with torch.no_grad():
    outputs = model(panel_embeddings, panel_mask)

contextualized_panels = outputs['contextualized_panels']  # (1, 6, 512)
strip_embedding       = outputs['strip_embedding']        # (1, 512)

print(f"Contextualized panels: {contextualized_panels.shape}")
print(f"Strip embedding:       {strip_embedding.shape}")

Example: Reading Order Scoring

python

with torch.no_grad():
    # order_matrix[0, i, j] = score indicating if panel i comes before panel j
    order_matrix = model.reading_order_head(panel_embeddings)  # (1, N, N)

# Compute sorting order based on average row scores
predicted_order = order_matrix[0].sum(dim=1).argsort(descending=True)
print(f"Predicted reading order: {predicted_order.tolist()}")

Example: Panel Picking (ComicsPAP-style)

python

# context: panels from the strip with one masked out
# candidates: 5 panel embeddings (1 correct, 4 distractors)
context_emb    = contextualized_panels[:, :5, :]   # (1, 5, 512)
candidate_embs = torch.randn(1, 5, 512).to(device) # (1, 5 candidates, 512)

with torch.no_grad():
    scores = model.panel_picking_head(context_emb.mean(dim=1), candidate_embs)
    predicted_idx = scores.argmax(dim=-1)
print(f"Predicted panel index: {predicted_idx.item()}")

Pipeline Position

Stage 1: Raw Comics → Panel crops + OCR text
Stage 2: CoSMo (PSS) → Narrative page classification
Stage 3: comic-panel-vlm-v1 → Multimodal panel embeddings (V + T + Composition) → (N, 512)
Stage 4: comic-strip-encoder-v1 → Contextualized panel + strip embeddings ← THIS MODEL
Stage 5: Storage & Query → Zarr store + semantic search

Intended Use

Narrative reranking: Stage 3 retrieves top-N candidates; Stage 4 strip embeddings rerank by sequence coherence.
Story-level similarity: Encode a query as a single panel → Stage 4 → compare strip embeddings across a corpus (story-level search, not panel-level).
Reading order auditing: Use the ReadingOrderHead pairwise matrix to verify or correct panel sequencing in digitised comics.
Narrative flow verification: Score a proposed page sequence for coherence using the closure heads.
Localisation/dialogue drift auditing: Use the TextClozeHead to flag pages where dialogue is likely misattributed or out of order.

Limitations

Fixed max sequence length: 16 panels per page (memory constraint at training time).
Discriminative only: Task heads require candidate sets; not a generative model.
Page-level only: Does not model multi-page narrative arcs.
Upstream dependency: Requires Stage 3 (comic-panel-vlm-v1) embeddings as input; raw images are not accepted directly.
No explicit character re-identification: The CharacterCoherenceHead scores visual consistency but does not track named characters across pages.

Performance Expectations

Task	Expected Accuracy	Random Baseline
Panel Picking	60–70%	20%
Visual Closure	55–65%	20%
Text Closure	50–60%	20%
Reading Order	75–85%	50%
Text-Cloze	50–60%	25%

Citation

If you use this model or the associated framework, please link back to the Comic Analysis GitHub Repository.

Related work this model is based on:

bibtex

@article{comicspap2025,
  title={ComicsPAP: A Panel-Aware Pipeline for Comic Understanding},
  year={2025},
  url={https://arxiv.org/abs/2503.08561}
}

@article{textcloze2024,
  title={Text-Cloze: Multimodal Dialogue Prediction in Comics},
  year={2024},
  url={https://arxiv.org/abs/2403.03719}
}