Variant	Encoder	Feature dim	Released weights	File
`hoptimus`	H-optimus-0 (ViT-g/14)	1536	Full (encoder + head)	`hoptimus/model.safetensors`
`resnet50`	ResNet-50	2048	Full (encoder + head)	`resnet50/model.safetensors`
`conch`	CONCH ViT-B/16	512	Head only	`conch/head.safetensors`

Repository contents

Path	Description
`modeling.py`	Model definitions — required to load any of the weights
`preprocess.py`	Whole-slide preprocessing: background removal, stain normalization, patching
`inference.py`	Command-line inference example
`sample_data/`	Example region and patches for testing (CC BY 4.0, see below)

About the CONCH variant

CONCH is distributed by MahmoodLab under CC-BY-NC-ND 4.0, which prohibits redistribution. Only the task head is released here — the encoder is not included and must be obtained by you.

Two important caveats:

You must request access to CONCH yourself (gated, non-commercial research use only) and install the conch package.
This head will not reproduce the paper's CONCH results. The reported numbers used an encoder that was fine-tuned jointly with the head; pairing this head with the original frozen CONCH encoder is a different model. It is provided for architecture reference and completeness only.

Predicted cell types

Group	Cell types
Lymphoid	CD4, CD8, B, Plasma
Myeloid	Macrophage, DC
Stromal	endothelial, Fibroblast
Tumor	malignant, NormalEpithelial, dead

Predictions are relative quantile values in [0, 1], not absolute cell counts.

Installation

pip install torch torchvision timm safetensors huggingface_hub pillow

For whole-slide preprocessing (preprocess.py), additionally:

pip install numpy tifffile scikit-image

For the CONCH variant, additionally:

pip install git+https://github.com/mahmoodlab/CONCH.git

Quick start

python

import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from modeling import load_model, build_transform, predict

REPO_ID = "mkkim032599/HNE2Space"
DEVICE  = "cuda" if torch.cuda.is_available() else "cpu"

weights = hf_hub_download(REPO_ID, filename="hoptimus/model.safetensors")

model     = load_model(weights, backbone="hoptimus", device=DEVICE)
transform = build_transform("hoptimus")

img = Image.open("patch.png").convert("RGB")
x   = transform(img).unsqueeze(0).to(DEVICE)

preds = predict(model, x)          # dict: cell type -> tensor in [0, 1]

for cell_type, value in preds.items():
    print(f"{cell_type}: {value.item():.4f}")

Or from the command line:

single image

python inference.py --input patch.png

one folder of patches -> one CSV

python inference.py --input patches/ --output results/

root of sample folders -> one CSV per sample

python inference.py --input root/ --output results/ --per-sample

ResNet-50 baseline

python

weights = hf_hub_download(REPO_ID, filename="resnet50/model.safetensors")
model     = load_model(weights, backbone="resnet50", device=DEVICE)
transform = build_transform("resnet50")

CONCH variant (encoder not included)

python

# 1. Request access at https://huggingface.co/MahmoodLab/CONCH and download
#    the encoder checkpoint yourself.
# 2. Pass its local path:

head = hf_hub_download(REPO_ID, filename="conch/head.safetensors")

model = load_model(
    head,
    backbone="conch",
    conch_checkpoint_path="/path/to/your/CONCH/pytorch_model.bin",
    device=DEVICE,
)
transform = build_transform("conch")

Batch inference

python

batch = torch.stack([transform(Image.open(p).convert("RGB")) for p in paths])
preds = predict(model, batch.to(DEVICE))   # each value has shape (B,)

Input requirements


Patch size	224 x 224
Resolution	approximately 0.27 microns per pixel
Stain	H&E

Normalization differs per encoder — always use build_transform(backbone) rather than ImageNet defaults:

Variant	Mean	Std
`hoptimus`	`(0.707223, 0.578729, 0.703617)`	`(0.211883, 0.230117, 0.177517)`
`conch`	`(0.48145466, 0.4578275, 0.40821073)`	`(0.26862954, 0.26130258, 0.27577711)`
`resnet50`	`(0.485, 0.456, 0.406)`	`(0.229, 0.224, 0.225)`

predict() converts raw outputs from log1p space back to quantile space. If you call model(x) directly, apply from_log1p() yourself.

Preprocessing

preprocess.py turns a whole-slide H&E image into model-ready patches:

tif -> background removal -> Reinhard normalization -> patches

1. Background removal. The slide is tiled into small blocks; blocks whose mean HSV saturation falls below a threshold are treated as background and zeroed out. This removes white glass and empty regions before normalization, so background pixels do not distort the colour statistics.

2. Reinhard colour normalization. Stain colour is matched to a reference slide in LAB space, channel by channel. Background (zero) pixels are excluded from the mean/std computation and left untouched. This reduces variation between scanners and staining batches. LAB conversion is chunked so large slides do not exhaust memory.

3. Patch extraction. Patches are cropped on a regular grid and kept only if they pass a saturation floor and a maximum background-fraction check.

Requirements

bash

pip install numpy pillow tifffile scikit-image
pip install h5py        # only for --save-h5

Usage

bash

# single slide
python preprocess.py --input slide.tif --target reference.tif --output out/

# directory of slides
python preprocess.py --input slides/ --pattern "**/Aligned-hne.tif" \
    --target reference.tif --output out/

# patches sized for the model
python preprocess.py --input slide.tif --target reference.tif --output out/ \
    --patch-size 256 --resize 224

--target is a reference slide whose stain appearance you want to match. Use the same reference across your whole cohort — switching references mid-way introduces exactly the batch effect this step is meant to remove.

Key options

Option	Default	Description
`--patch-size`	`256`	Crop size in pixels
`--stride`	= patch size	Step between patches; smaller values overlap
`--resize`	none	Resize patches (use `224` for the model)
`--bg-sat-threshold`	`0.10`	Saturation below which a block is background
`--patch-sat-threshold`	`0.10`	Minimum mean saturation to keep a patch
`--black-threshold`	`0.10`	Max fraction of background pixels per patch
`--no-background-removal`	off	Skip stage 1
`--no-normalize`	off	Skip stage 2
`--save-intermediate`	off	Also write the normalized slide and a JPEG preview
`--save-h5`	off	Additionally bundle patches into an HDF5 file

Thresholds are starting points, not tuned values — inspect the output with --save-intermediate and adjust them for your slides.

Resolution

The encoder expects roughly 0.27 microns per pixel. preprocess.py crops in pixel units and does not read scanner MPP metadata, so choose --patch-size and --resize to match your source resolution. For example, a slide at 0.25 um/px needs a 448-pixel crop resized to 224 to land at ~0.5 um/px.

Sample data

sample_data/ contains a small H&E region and a few prepared patches so you can verify the pipeline end to end without downloading a full slide.

bash

# run the model on a prepared patch
python inference.py sample_data/patches/patch_00000.png --backbone hoptimus

# or start from the region crop
python preprocess.py --input sample_data/region.tif \
    --target sample_data/region.tif --output my_patches/ --resize 224

These images derive from a publicly available 10x Genomics dataset licensed under CC BY 4.0, cropped and tiled for this repository. See sample_data/README.md for the dataset link and full attribution.

The sample images are distributed under CC BY 4.0 (inherited from the source dataset), not under the Apache-2.0 licence covering the code and weights.

Architecture

encoder                       -> feature_dim  (1536 / 2048 / 512)
  shared trunk                -> 256          (via 512 for hoptimus)
  lineage branches      256   -> 128          (lymphoid / myeloid / stromal / tumor)
  per-task gating       128   -> 128          (sigmoid attention gate)
  regression heads      128   -> 1            (one per cell type)

Each cell type is routed through its lineage branch, then through a task-specific gate before its regression head.

If a checkpoint was trained with different architecture settings, pass them explicitly — the defaults must match how the weights were trained:

model = load_model(weights, backbone="resnet50", input_proj_dim=None)

Limitations and intended use

Research use only. Not a medical device; not validated for clinical or diagnostic use. Performance on unseen tissue types, scanners, or staining protocols may degrade.
Outputs are relative quantile values, not absolute cell counts or densities.
Accuracy varies by cell type; rarer populations are predicted less reliably. Consult the evaluation table before relying on any individual cell type.

License and attribution

The code and the hoptimus / resnet50 weights are released under the Apache License 2.0.

The hoptimus weights are a derivative work of H-optimus-0 (Bioptimus), also Apache-2.0. The encoder weights have been modified (fine-tuned) from the original release. The original copyright notice is retained in NOTICE.

H-optimus-0: https://huggingface.co/bioptimus/H-optimus-0
ResNet-50 (torchvision, ImageNet-pretrained initialisation)

Citation

bibtex

@software{hoptimus0,
  author = {Saillard, Charlie and Jenatton, Rodolphe and Llinares-L{\'o}pez, Felipe
            and Mariet, Zelda and Cahan{\'e}, David and Durand, Eric and Vert, Jean-Philippe},
  title  = {H-optimus-0},
  year   = {2024},
  url    = {https://github.com/bioptimus/releases/tree/main/models/h-optimus/v0}
}

@article{lu2024conch,
  title   = {A visual-language foundation model for computational pathology},
  author  = {Lu, Ming Y and Chen, Bowen and Williamson, Drew F K and others},
  journal = {Nature Medicine},
  year    = {2024}
}