publications

2026

1. 

   BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks

   Ivan Sabolić, Marin Oršić, Josip Šarić, and 1 more author

   In International Conference on Machine Learning, 2026

   Abs arXiv Bib PDF Poster

   Supervised fine-tuning is the predominant approach for adapting autoregressive vision-language models to downstream tasks. Recent work has shown that this paradigm is highly vulnerable to backdoor attacks, and that existing defenses are ineffective in open-ended generation settings. In response, we propose BYORn, a backdoor-robust fine-tuning framework motivated by the observation that poisoned target responses are often semantically implausible given the corresponding image-text inputs and a pretrained model. BYORn identifies such misaligned responses and dynamically replaces them with alternative responses generated by the model, thereby breaking the correlation between triggers and target outputs. The resulting objective gradient corresponds to the gradient of the empirical estimate of the population risk upper bound over the clean data distribution. Empirically, BYORn consistently improves robustness to backdoor attacks while preserving clean-task performance, establishing a new trade-off frontier between generalization and attack success rate. Finally, we demonstrate that BYORn remains effective against adaptive attacks specifically designed to circumvent the proposed defense.

   @inproceedings{sabolic2026byorn,
     title = {BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attacks},
     author = {Saboli{\'c}, Ivan and Or{\v{s}}i{\'c}, Marin and {\v{S}}ari{\'c}, Josip and Lon{\v{c}}ari{\'c}, Sven},
     booktitle = {International Conference on Machine Learning},
     year = {2026},
   }

2025

1. 

   Seal Your Backdoor with Variational Defense

   Ivan Sabolić, Matej Grcić, and Siniša Šegvić

   International Conference on Computer Vision, 2025

   Abs Bib PDF Code

   We propose VIBE, a model-agnostic framework that trains classifiers resilient to backdoor attacks. The key concept behind our approach is to treat malicious inputs and corrupted labels from the training dataset as observed random variables, while the actual clean labels are latent. VIBE then recovers the corresponding latent clean label posterior through variational inference. The resulting training procedure follows the expectation-maximization (EM) algorithm. The E-step infers the clean pseudolabels by solving an entropy-regularized optimal transport problem, while the M-step updates the classifier parameters via gradient descent. Being modular, VIBE can seamlessly integrate with recent advancements in self-supervised representation learning, which enhance its ability to resist backdoor attacks. We experimentally validate the method effectiveness against contemporary backdoor attacks on standard datasets, a large-scale setup with 1k classes, and a dataset poisoned with multiple attacks. VIBE consistently outperforms previous defenses across all tested scenarios.

   @article{sabolic2025seal,
     title = {Seal Your Backdoor with Variational Defense},
     author = {Saboli{\'c}, Ivan and Grci{\'c}, Matej and {\v{S}}egvi{\'c}, Sini{\v{s}}a},
     journal = {International Conference on Computer Vision},
     year = {2025},
   }

2024

1. 

   Backdoor Defense through Self-Supervised and Generative Learning

   Ivan Sabolic, Ivan Grubišić, and Siniša Šegvić

   In 35th British Machine Vision Conference 2024, BMVC 2024, Glasgow, UK, November 25-28, 2024, 2024

   Abs Bib PDF Code

   Backdoor attacks change a small portion of training data by introducing hand-crafted triggers and rewiring the corresponding labels towards a desired target class. Training on such data injects a backdoor which causes malicious inference in selected test samples. Most defenses mitigate such attacks through various modifications of the discriminative learning procedure. In contrast, this paper explores an approach based on generative modelling of per-class distributions in a self-supervised representation space. Interestingly, these representations get either preserved or heavily disturbed under recent backdoor attacks. In both cases, we find that per-class generative models allow to detect poisoned data and cleanse the dataset. Experiments show that training on cleansed dataset greatly reduces the attack success rate and retains the accuracy on benign inputs.

   @inproceedings{Sabolic_2024_BMVC,
     author = {Sabolic, Ivan and Grubišić, Ivan and Šegvić, Siniša},
     title = {Backdoor Defense through Self-Supervised and Generative Learning},
     booktitle = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow, UK, November 25-28, 2024},
     publisher = {BMVA},
     year = {2024},
     url = {https://papers.bmvc2024.org/0346.pdf},
   }

2023

1. 

   Computational Color Constancy-Based Backdoor Attacks

   Donik Vršnak, Ivan Sabolić, Marko Subašić, and 1 more author

   In 2023 International Symposium on Image and Signal Processing and Analysis (ISPA), 2023

   Abs Bib PDF

   Deep neural networks (DNNs) have become an integral part of many computer vision tasks. However, training complex neural networks requires a large amount of computational resources. Therefore, many users outsource training to third parties. This introduces an attack vector for backdoor attacks. These attacks are described as attacks in which the neural network behaves as expected for benign inputs but acts maliciously when a backdoor trigger is present in the input. Triggers are small, preferably stealthy additions to the input. However, most of these triggers are based on the additive model, i.e., the trigger is simply added onto the image. Furthermore, optimized triggers are artificial, which means that it is difficult or impossible to reproduce them in the real-world, making them impractical to use in a real-world setting. In this work, we present a novel way of trigger injection for the classification problem. It is based on the von Kries model for image color correction, a frequently used component in all image processing pipelines. Our trigger uses a multiplicative rather than an additive model. This makes it harder to detect the injection by defensive methods. Second, the trigger is based on real-world phenomena of changing illumination. Finally, it can be made harder to spot by a human observer, when compared to some additive triggers. We test the performance of our attack strategy against various defense methods on several frequently used datasets, and achieve excellent results. Furthermore, we show that the malicious behavior of models trained on artificially colored images can be activated in real-world scenarios, further increasing the usefulness of our attack strategy.

   @inproceedings{10278694,
     author = {Vršnak, Donik and Sabolić, Ivan and Subašić, Marko and Lončarić, Sven},
     booktitle = {2023 International Symposium on Image and Signal Processing and Analysis (ISPA)},
     title = {Computational Color Constancy-Based Backdoor Attacks},
     year = {2023},
     volume = {},
     number = {},
     pages = {1-6},
     keywords = {Training;Computer vision;Additives;Image color analysis;Computational modeling;Pipelines;Lighting},
   }