NV-Segment-CTMR is a specialized foundation model for 3D medical image segmentation that excels at accurate, adaptable, automatic segmentation across anatomies and modalities, including computed tomography (CT) and magnetic resonance (MR) imaging. NV-Segment-CTMR adapts to varying conditions and anatomical regions, enabling comprehensive automated annotation workflows.
At the core of NV-Segment-CTMR are two automated workflows. Segment Everything enables whole-body exploration, which is crucial for understanding complex diseases affecting multiple organs and for holistic treatment planning. Segment by Class provides detailed sectional views based on specific classes, supporting targeted disease analysis or organ mapping, such as tumor identification in critical organs.
For running the pipeline, NV-Segment-CTMR requires at least one prompt for segmentation. It supports label prompt, which is the index of the class for automatic segmentation. NV-Segment-CTMR does not support point based interactive segmentation. For interactive model, please refer to VISTA3D
Here is a code snippet to showcase how to execute inference with this model.
The inputs must include the key image which contain the absolute path to the nii image file, and includes prompt keys of label_prompt.
The label_prompt is a list of length B, which can perform B foreground objects segmentation, e.g. [2,3,4,5]. The full list of label definition is in metadata.json.
If no prompt is provided, user can use modality to use predefined class indices. Supported modality includes CT_BODY, MRI_BODY, MRI_BRAIN.
Note: For brain structure segmentation, current model only support standard brain T1 images. The brain T1 images must be preprocessed with skull stripping and normalization. Follow https://github.com/junyuchen245/MIR/tree/main/tutorials/brain_MRI_preprocessing to process the brain images
Code License
This project includes code licensed under the Apache License 2.0. You may obtain a copy of the License at
Medical researchers, AI developers, and healthcare institutions are expected to use this system to perform automated medical image segmentation, conduct multi-organ analysis, and accelerate annotation workflows in research applications.
[1] He, Yufan, et al. "VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging." arXiv preprint arXiv:2406.05285. 2024. https://arxiv.org/abs/2406.05285
Model Architecture:
Architecture Type: Transformer
Network Architecture: SAM-like architecture for 3D medical imaging segmentation
This model was developed from scratch using MONAI components.
Number of model parameters: 218M
Input:
Input Type(s): Image
Input Format(s): Neuroimaging Informatics Technology Initiative (NIfTI)
Input Parameters: Three-Dimensional (3D)
Other Properties Related to Input: Supports both computed tomography (CT) and magnetic resonance (MR) imaging modalities. It also supports optional class information for targeted segmentation workflows.
Input Modalities:
CT Images: 3D computed tomography volumes
MR Images: 3D magnetic resonance volumes
Class Selection: Optional class indices for targeted segmentation workflows
Output:
Output Type(s): Image
Output Format: Neuroimaging Informatics Technology Initiative (NIfTI)
Output Parameters: Three-Dimensional (3D)
Other Properties Related to Output: Segmentation masks with up to 345+ anatomical classes, providing comprehensive organ and tissue delineation for medical imaging analysis.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (GPU cores) and software frameworks (CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
0.1 - Initial release version for 3D medical imaging segmentation with multi-modality support
Training, Testing, and Evaluation Datasets:
Dataset Overview:
Total Size: ~31k
Total Number of Datasets: 32 datasets
Public datasets from multiple scanner types were processed to create standardized 3D medical imaging volumes with expert-validated anatomical segmentation masks across diverse anatomical regions and pathological conditions. The data processing pipeline ensured consistent voxel spacing, standardized orientations, and validated anatomical segmentations.
Training Dataset:
Data Modality:
Image
Image Training Data Size:
Less than a Million Images
Data Collection Method by dataset:
Hybrid: Human, Automatic/Sensors
Labeling Method by dataset:
Hybrid: Human, Automatic/Sensors
Testing Dataset:
Data Collection Method by dataset:
Hybrid: Human, Automatic/Sensors
Labeling Method by dataset:
Hybrid: Human, Automatic/Sensors
Evaluation Dataset:
Data Collection Method by dataset:
Hybrid: Human, Automatic/Sensors
Labeling Method by dataset:
Hybrid: Human, Automatic/Sensors
Inference:
Acceleration Engine: PyTorch
Test Hardware:
A100
H100
Additional Information:
Available Anatomical Classes (345+ total):
NV-Segment-CTMR supports comprehensive anatomical segmentation with the following categories:
Appendicular: Bilateral long bones, joints, and extremities (87-96)
Detailed Brain Segmentation:
Comprehensive brain parcellation including ventricles, cortical regions, subcortical structures, and specialized brain areas (214-345) based on neuroanatomical atlases.
Note: We recommend the NV-Segment-CT model for better tumor performance.
Specialized Regions:
Head and neck: detailed facial structures, sensory organs, and cranial anatomy (172-213)
Cardiac: heart chambers, major vessels, and cardiac-specific structures (108, 149-155)
Reproductive: prostate zones (118, 147-148), uterocervix (161), gonads (160)
Complete numerical mapping and deprecated classes available in model documentation.
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.
Please report model quality, risk, security vulnerabilities or concerns here.
