PP-SAM: Perturbed Prompts for Robust Adaptation of Segment Anything Model for Polyp Segmentation

The Segment Anything Model (SAM), originally designed for general-purposesegmentation tasks, has been used recently for polyp segmentation. Nonetheless,fine-tuning SAM with data from new imaging centers or clinics poses significantchallenges. This is because this necessitates the creation of an expensive andtime-intensive annotated dataset, along with the potential for variability inuser prompts during inference. To address these issues, we propose a robustfine-tuning technique, PP-SAM, that allows SAM to adapt to the polypsegmentation task with limited images. To this end, we utilize variableperturbed bounding box prompts (BBP) to enrich the learning context and enhancethe model's robustness to BBP perturbations during inference. Rigorousexperiments on polyp segmentation benchmarks reveal that our variable BBPperturbation significantly improves model resilience. Notably, on Kvasir,1-shot fine-tuning boosts the DICE score by 20BBP perturbations during inference, respectively. Moreover, our experimentsshow that 1-shot, 5-shot, and 10-shot PP-SAM with 50-pixel perturbations duringinference outperform a recent state-of-the-art (SOTA) polyp segmentation methodby 26applicability of our PP-SAM for other medical imaging tasks with limitedsamples. Our implementation is available at https://github.com/SLDGroup/PP-SAM.