You can deploy this model and start earning money today!

Jpcosta90/
cosdoc-cross-modal

Name: cosdoc-cross-modal
Availability: InStock

document-understanding

Views

No views yet

CosDoc — Cross-Modal Pooler

CosDoc is a visual document embedding model trained with supervised metric learning and hard-example selection via a Reinforcement Learning professor network.

Pooler variant: Cross-Modal Pooler — Bidirectional cross-modal attention pooler (visual ↔ text).

Architecture

Component	Value
Backbone	InternVL3-2B (`OpenGVLab/InternVL3-2B`)
Cut layer	27
Pooler	cross_modal (num_queries=1)
Embedding dim	1536
Loss	Sub-Center CosFace (m=0.35, s=32, k=3)
Embedding prompt	`<image> Analyze this document`

Performance (LA-CDIP, full validation pairs)

Dataset	EER
LA-CDIP (5-fold CV)	1.25%

Source run: Sprint3b_S0_subcenter_cosface_seed42_noinit_cross_modal_fase2_profOFF_E5

Usage

python

import torch
from huggingface_hub import hf_hub_download
from cavl_doc.models.backbone_loader import load_model
from cavl_doc.models.modeling_cavl import build_cavl_model

device = "cuda" if torch.cuda.is_available() else "cpu"

# Download fine-tuned weights
ckpt_path = hf_hub_download(repo_id="Jpcosta90/cosdoc-cross-modal", filename="best_model.pt")
ckpt = torch.load(ckpt_path, map_location=device, weights_only=False)
cfg  = ckpt["config"]

backbone, _, tokenizer, _, _ = load_model("InternVL3-2B")
model = build_cavl_model(
    backbone=backbone,
    cut_layer=cfg["cut_layer"],
    pooler_type=cfg["pooler_type"],
    num_queries=cfg.get("num_queries", 1),
)
model.pool.load_state_dict(ckpt["siam_pool"])
model.head.load_state_dict(ckpt["siam_head"])
model.eval().to(device)

Citation

bibtex

@misc{cosdoc2026,
  title  = {CosDoc: Cosine-Margin Document Embeddings with RL-guided Hard Mining},
  author = {Costa, João Paulo},
  year   = {2026},
  url    = {https://huggingface.co/Jpcosta90/cosdoc-cross-modal}
}