CancerTranscriptome-Mini-48M is a lightweight derivative of BulkFormer, designed to learn cancer-specific transcriptomic structure from large-scale bulk RNA-seq.
It combines GCN-based gene graph propagation, Rotary Expression Embeddings (REE), local bin-wise Performer attention, and global Performer attention into a single unified encoder.

This model is a proof-of-concept designed for research, experimentation, and rapid iteration on BulkFormer-style architectures applied to cancer transcriptomes.

🔬 Origin & References

Primary Reference (BulkFormer)

Boming Kang, Rui Fan, Meizheng Yi, Chunmei Cui, Qinghua Cui.
“A large-scale foundation model for bulk transcriptomes.”
bioRxiv (2025).
doi: https://doi.org/10.1101/2025.06.11.659222

This Model (CancerTranscriptome-Mini-48M)

A compact re-implementation based on the BulkFormer architecture, adapted for cancer-only bulk RNA-seq and simplified for accessibility and compute efficiency.
Source Code: https://github.com/alwalt/BioFM

📊 Data Source

All training samples originate from the ARCHS4 Human RNA-seq v2.5 public repository:

ARCHS4 Reference:
Lachmann A., Torre D., Keenan A.B., Jagodnik K.M., et al.
“Massive mining of publicly available RNA-seq data from human and mouse.”
Nature Communications 9, 1366 (2018).
Dataset: https://maayanlab.cloud/archs4/

Filtering Procedure

Loaded all human bulk RNA-seq metadata from ARCHS4 v2.5 HDF5
Selected samples matching:
cancer | tumor | carcinoma | leukemia | lymphoma | melanoma | glioma
Removed samples lacking clear disease annotations
Used ARCHS4 log-TPM matrices (gene × sample)
Final dataset: ~76k cancer samples, 19,357 genes

No private, clinical, controlled-access, or proprietary data were used.

🧠 Model Architecture (Summary)

CancerTranscriptome-Mini-48M includes:

1. Gene Identity Embeddings

Precomputed ESM2 embeddings for each protein-coding gene
Projected into model dimension (320)

2. Rotary Expression Embeddings (REE)

Deterministic sinusoidal continuous-value embedding
Masked positions zeroed (mask token = –10)

3. Graph Neural Network Layer

GCNConv (Kipf & Welling) applied on a curated gene-gene graph
Injects biological prior knowledge

4. Expression Binning

Learnable importance scores sort genes
Genes divided into 10 bins
Each bin receives its own local Performer attention

5. Global Performer Attention

2 stacked Performer layers across all genes

6. Prediction Head

MLP → scalar value per gene
Used for masked-expression reconstruction

Total parameters: 48,336,162 (~48M)

🎯 Intended Use

This model produces context-aware gene embeddings for downstream cancer transcriptomic tasks:

Tumor subtype prediction
Drug response modeling
Immune infiltration scoring
Survival / risk modeling
Gene expression imputation
Dimensionality reduction
Transfer learning to TCGA, CCLE, DepMap, GEO tumor datasets

🚀 How to Use

Download & run:

python

import torch
from model import BulkFormer   # from this repo
import safetensors.torch as st

# Load model + weights
model = BulkFormer(
    dim=320,
    graph=torch.load("edge_index.pt"),   # provide your graph
    gene_emb=torch.load("esm2_gene_emb.pt"),
    gene_length=19357,
    bin_head=8,
    full_head=4,
    bins=10,
    gb_repeat=1,
    p_repeat=2
)

state = st.load_file("model.safetensors")
model.load_state_dict(state)
model.eval()

# Example input: 19,357-gene log-TPM vector
x = torch.randn(1, 19357)

with torch.no_grad():
    out = model(x)

print(out.shape)  # [1, 19357]

alvawalt/CancerTranscriptome-Mini-48M

🧬 CancerTranscriptome-Mini-48M