Adversarially Robust Cross-Modal Medical Image Retrieval via Self-Supervised Deep Hashing and Large Language Model Agents
Keywords:
adversarial robustness, cross-modal retrieval, medical imaging, self-supervised learning, deep hashing, large language model agents, healthcare AI governanceAbstract
Cross-modal medical image retrieval supports clinical decision-making by enabling semantically grounded queries across heterogeneous data sources, yet its deployment in real-world settings faces substantial challenges from adversarial perturbations, modality gaps, and governance demands. This paper advances a system-level framework that integrates self-supervised deep hashing with large language model (LLM) agents to achieve adversarially robust, semantically precise retrieval. The architecture couples a self-supervised hashing backbone that learns modality-agnostic binary codes from unpaired radiological reports and imaging studies with an LLM-based semantic mediator that reformulates queries, validates retrieved candidates, and injects domain constraints. A threat model encompassing modality-specific gradient-based perturbations, linguistic prompt injection, and distributional drift is formally characterized. Defense mechanisms are woven throughout the retrieval pipeline, including adversarial hashing via margin-scalable constraints, randomized smoothing for certified robustness, and prompt sanitization layers within the LLM agent. The discussion emphasizes structural trade-offs among retrieval latency, bit-width efficiency, and robustness guarantees. Governance implications are analyzed with regard to fairness across patient subpopulations, audit trails for retrieval decisions, energy sustainability of dual-stage architectures, and compliance with evolving regulatory frameworks for AI-enabled medical devices. By treating robustness and governance not as afterthoughts but as design constraints embedded within the self-supervised training loop and the agent orchestration, the framework aims to bridge the gap between laboratory validation and trustworthy clinical deployment. This systems-oriented synthesis highlights open challenges including continual adaptation under domain shift, scalable federated hash learning, and the need for standardized benchmark protocols that jointly evaluate retrieval accuracy, adversarial resilience, and fairness metrics in cross-modal medical search.
References
1. Müller, H., Michoux, N., Bandon, D., & Geissbuhler, A. (2004). A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. International Journal of Medical Informatics, 73(1), 1–23.
2. Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019). Adversarial attacks on medical machine learning. Science, 363(6433), 1287–1289.
3. Liu, H., Wang, R., Shan, S., & Chen, X. (2016). Deep supervised hashing for fast image retrieval. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2064–2072.
4. Azizi, S., Mustafa, B., Ryan, F., Beaver, Z., Freyberg, J., Deaton, J., Loh, A., Kornblith, S., Chen, T., & Norouzi, M. (2021). Big self-supervised models advance medical image classification. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3478–3488.
5. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner-Fushman, D., … Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172–180.
6. Char, D. S., Shah, N. H., & Magnus, D. (2018). Implementing machine learning in health care—addressing ethical challenges. New England Journal of Medicine, 378(11), 981–983.
7. Jiang, Q. Y., & Li, W. J. (2017). Deep cross-modal hashing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3232–3240.
8. Hu, S. (2026). Research on Security Enhancement Methods for Adversarial Robust Large Language Model Intelligent Agents for Medical Decision-Making Tasks. arXiv preprint arXiv:2605.08257.
9. Yu, Z., Wu, S., Dou, Z., & Bakker, E. M. (2022). Deep hashing with self-supervised asymmetric semantic excavation and margin-scalable constraint. Neurocomputing, 483, 87-104.
10. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. Proceedings of the International Conference on Learning Representations (ICLR).
11. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
12. Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H. R., Albarqouni, S., Bakas, S., Galtier, M. N., Landman, B. A., Maier-Hein, K., Ourselin, S., Sheller, M., Summers, R. M., Trask, A., Xu, D., Baust, M., & Cardoso, M. J. (2020). The future of digital health with federated learning. NPJ Digital Medicine, 3(1), 119.
13. Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning (ICML), 1597–1607.
14. Wang, J., Zhang, T., Song, J., Sebe, N., & Shen, H. T. (2018). A survey on learning to hash. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 769–790.
15. Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Flores, M. G., Zhang, Y., Magoc, T., Harle, C. A., Lipori, G., Mitchell, D. A., Hogan, W. R., Shenkman, E. A., Bian, J., & Wu, Y. (2023). Large language models in medicine: the potentials and pitfalls. arXiv preprint arXiv:2309.10980.
16. Cohen, J., Rosenfeld, E., & Kolter, Z. (2019). Certified adversarial robustness via randomized smoothing. Proceedings of the International Conference on Machine Learning (ICML), 1310–1320.
17. Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020). The ethics of AI in healthcare: A mapping review. Social Science & Medicine, 260, 113172.
18. Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 3645–3650.
19. Eslami, S., de Melo, G., & Meinel, C. (2023). MedCLIP: Contrastive learning from unpaired medical images and text. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 3876–3887.
20. Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B. A., Haque, I. S., Beery, S. M., Leskovec, J., Kundaje, A., … Liang, P. (2021). WILDS: A benchmark of in-the-wild distribution shifts. Proceedings of the International Conference on Machine Learning (ICML), 5637–5664.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 International Journal of Clinical and Translational Medicine
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
