Segmentation by registration-enabled SAM prompt engineering using five reference images
0
Sign in to get full access
Overview
- New method for image segmentation using a Segment Anything Model (SAM) with reference images
- Aims to improve segmentation accuracy by using 5 reference images to guide the SAM prompt engineering
- Tested on cartilage segmentation in MRI scans
Plain English Explanation
This research proposes a new way to use the Segment Anything Model (SAM), a powerful AI tool for segmenting objects in images. The key idea is to use 5 reference images to help guide the prompts (instructions) given to the SAM, with the goal of improving the accuracy of the segmentation, especially for complex structures like cartilage in medical images.
The researchers tested this approach on MRI scans of cartilage, and found that by using the 5 reference images, they were able to get more accurate segmentations compared to using the SAM alone without any reference images. This is an important advancement, as being able to accurately segment medical images is crucial for many clinical applications.
Technical Explanation
The researchers developed a registration-enabled SAM prompt engineering approach that leverages 5 reference images to guide the prompts given to the SAM model. The reference images are first registered (aligned) to the target image, and then the SAM prompts are engineered based on the aligned reference images.
This approach was tested on the task of cartilage segmentation in MRI scans. The researchers found that using the 5 reference images led to significantly better segmentation performance compared to using the SAM model alone without any reference images. The improvements were quantified using standard segmentation metrics like Dice score.
Critical Analysis
The researchers acknowledge that their approach relies on having access to suitable reference images, which may not always be available in practical settings. Additionally, the registration step adds computational complexity and may not always be perfectly accurate, which could impact the final segmentation results.
While the results are promising, further research is needed to fully understand the limitations and generalizability of this approach. Applying it to a wider range of medical imaging modalities and anatomical structures would help validate its broader applicability.
Conclusion
This research presents a novel way to leverage the Segment Anything Model (SAM) for improved image segmentation, particularly in the medical imaging domain. By using 5 reference images to guide the prompt engineering, the researchers were able to achieve more accurate cartilage segmentation in MRI scans compared to using the SAM model alone.
This work highlights the potential of using reference-guided prompt engineering to enhance the capabilities of powerful AI models like SAM, and could have important implications for various clinical applications that rely on accurate image segmentation.
This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!
Related Papers
0
Segmentation by registration-enabled SAM prompt engineering using five reference images
Yaxi Chen, Aleksandra Ivanova, Shaheer U. Saeed, Rikin Hargunani, Jie Huang, Chaozong Liu, Yipeng Hu
The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at: https://github.com/chrissyinreallife/KneeSegmentWithSAM.git
Read more7/26/2024
0
Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2
Andrew Seohwan Yu, Mohsen Hariri, Xuecen Zhang, Mingrui Yang, Vipin Chaudhary, Xiaojuan Li
Intelligent medical image segmentation methods are rapidly evolving and being increasingly applied, yet they face the challenge of domain transfer, where algorithm performance degrades due to different data distributions between source and target domains. To address this, we introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2 (SAM2), a general-purpose segmentation model designed to accept prompts and retain memory across frames of a video. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning, accurately delineating structures in knee MRI scans using only a single prompt. Our experiments on the Osteoarthritis Initiative Zuse Institute Berlin (OAI-ZIB) dataset reveal that SAM2 achieves high accuracy on 3D knee bone segmentation, with a testing Dice similarity coefficient of 0.9643 on tibia. We also present results generated using different SAM2 model sizes, different prompt schemes, as well as comparative results from the SAM1 model deployed on the same dataset. This breakthrough has the potential to revolutionize medical image analysis by providing a scalable, cost-effective solution for automated segmentation, paving the way for broader clinical applications and streamlined workflows.
Read more8/12/2024
🛠️
0
AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation
Chengyin Li, Prashant Khanduri, Yao Qiang, Rafi Ibn Sultan, Indrin Chetty, Dongxiao Zhu
Segment Anything Model (SAM) is one of the pioneering prompt-based foundation models for image segmentation and has been rapidly adopted for various medical imaging applications. However, in clinical settings, creating effective prompts is notably challenging and time-consuming, requiring the expertise of domain specialists such as physicians. This requirement significantly diminishes SAM's primary advantage - its interactive capability with end users - in medical applications. Moreover, recent studies have indicated that SAM, originally designed for 2D natural images, performs sub optimally on 3D medical image segmentation tasks. This subpar performance is attributed to the domain gaps between natural and medical images and the disparities in spatial arrangements between 2D and 3D images, particularly in multi-organ segmentation applications. To overcome these challenges, we present a novel technique termed AutoProSAM. This method automates 3D multi-organ CT-based segmentation by leveraging SAM's foundational model capabilities without relying on domain experts for prompts. The approach utilizes parameter-efficient adaptation techniques to adapt SAM for 3D medical imagery and incorporates an effective automatic prompt learning paradigm specific to this domain. By eliminating the need for manual prompts, it enhances SAM's capabilities for 3D medical image segmentation and achieves state-of-the-art (SOTA) performance in CT-based multi-organ segmentation tasks.
Read more6/28/2024
0
Segment anything model 2: an application to 2D and 3D medical images
Haoyu Dong, Hanxue Gu, Yaqian Chen, Jichen Yang, Yuwen Chen, Maciej A. Mazurowski
Segment Anything Model (SAM) has gained significant attention because of its ability to segment various objects in images given a prompt. The recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field. In this paper, we extensively evaluate SAM 2's ability to segment both 2D and 3D medical images by first collecting 21 medical imaging datasets, including surgical videos, common 3D modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) as well as 2D modalities such as X-ray and ultrasound. Two evaluation settings of SAM 2 are considered: (1) multi-frame 3D segmentation, where prompts are provided to one or multiple slice(s) selected from the volume, and (2) single-frame 2D segmentation, where prompts are provided to each slice. The former only applies to videos and 3D modalities, while the latter applies to all datasets. Our results show that SAM 2 exhibits similar performance as SAM under single-frame 2D segmentation, and has variable performance under multi-frame 3D segmentation depending on the choices of slices to annotate, the direction of the propagation, the predictions utilized during the propagation, etc. We believe our work enhances the understanding of SAM 2's behavior in the medical field and provides directions for future work in adapting SAM 2 to this domain. Our code is available at: https://github.com/mazurowski-lab/segment-anything2-medical-evaluation.
Read more8/23/2024