Parameter efficient fine-tunning of foundation model to facilitate tumor response prediction for ovarian cancer patients

Abstract

Background In clinical practice, it is critically important to predict the tumor response of chemotherapy at an early stage. However, the performance of existing clinical markers cannot achieve satisfactory accuracy and reliability. To overcome this limitation, in this study we proposed a foundation model based approach to improve prediction performance. Methods We adopted a CLIP (Contrastive language-image pretraining) based foundation model, which contains a total of 12 transformer layers for both image and textual encoders. This model has been extensively pretrained on a non-medical image-text dataset. Meanwhile, a parameter efficient low rank adaptor (LoRA) was incorporated into the transformer layers for fine-tuning purpose. The adaptors were inserted at various layers of both encoders, and their corresponding performances were evaluated and compared. The experiments were conducted on a retrospective dataset containing a total of 182 advanced stage ovarian cancer cases, among which 124 were responders and 58 were non-responders. Results The best performance was achieved by adding LoRA adaptors on the lowest 5 transformer layers for both image and textual encoders, which yields an AUC (area under the receiver operating characteristic curve) of 0.785 ± 0.039 and an ACC (accuracy) of 0.780 ± 0.032. As a comparison, the conventional transfer learning strategy fine-tuned ResNet and ViT models achieved AUCs of 0.754 ± 0.089 and 0.707 ± 0.079, respectively. Conclusion This study initially demonstrates the feasibility of parameter efficient training with a foundation model for chemotherapy response prediction, highlighting its potential to support clinical decision making.