SIAMESE: Stealing Fine-Tuned Visual Foundation Models via Diversified Prompting

Document Type

Conference Proceeding

Publication Date

12-3-2025

Abstract

Visual foundation models, characterized by their robust generalization and adaptability, serve as the basis for a wide array of downstream tasks. When fine-tuned for specific tasks, these models encapsulate confidential and valuable task-specific knowledge, making them prime targets for model stealing (MS) attacks. While recent efforts have exposed MS threats in practical scenarios such as data-free and hard-label contexts, these attacks predominantly target traditional victim models trained from scratch. Fine-tuned visual foundation models, pre-trained on vast and diverse datasets and then fine-tuned on downstream tasks, present significant challenges for traditional MS attacks to extract task-specific knowledge. In this paper, we introduce an innovative MS attack, named SIAMESE, to steal fine-tuned visual foundation models under black-box, data-free, and hard-label settings. The core approach of SIAMESE involves constructing a stolen model using a foundation model that is efficiently and concurrently fine-tuned with multiple diversified soft prompts. To integrate the knowledge derived from these prompts, we propose a novel and tractable loss function that analyzes the output distributions while enforcing orthogonality among the prompts to minimize interference. Additionally, a unique alignment module enhances SIAMESE by synchronizing interpretations between the victim and stolen models. Extensive experiments validate that SIAMESE outperforms state-of-the-art baseline attacks over 10% in accuracy, exposing the heightened vulnerability of fine-tuned visual foundation models to MS threats.

Publication Title

Sec 2025 Proceedings of the 2025 10th ACM IEEE Symposium on Edge Computing

ISBN

[9798400722387]

Link to Full Text

Share

COinS