Evidence-Calibrated Biomedical Decision Support Using Retrieval-Augmented Large Language Models for Clinical Literature Interpretation and Uncertainty-Aware Diagnosis Prediction

Authors

  • Bahuraksa Adriansyah Department of Computer Science, Colorado State University, Fort Collins, CO, USA.
  • Milosz Galos Department of Computer Science, University of Houston, Houston, TX, USA.
  • Aadi Bhandari Department of Computer Science, Binghamton University, Binghamton, NY, USA.
  • Eshana Sibal Department of Electrical Engineering and Computer Science, University of Missouri, Columbia, MO, USA.

Keywords:

retrieval-augmented generation; clinical literature interpretation; uncertainty quantification; calibration; diagnosis prediction; evidence synthesis

Abstract

Biomedical decision support requires both faithful interpretation of clinical literature and calibrated diagnostic prediction, yet retrieval-augmented language models often treat retrieved text as uniformly reliable and produce confidence scores that are poorly aligned with evidence strength. This manuscript develops an evidence-calibrated framework that couples retrieval-augmented large language models with uncertainty-aware diagnosis prediction. The framework represents clinical queries and patient features jointly, scores documents by semantic relevance, evidence level, recency, and risk of bias, and converts retrieval scores into a soft evidence distribution. Generated literature interpretations are decomposed into claims whose evidential support is measured against retrieved spans and study-quality metadata. Diagnostic probabilities are then obtained by integrating over retrieval subsets and model variants, allowing total uncertainty to be separated into aleatoric and epistemic components. A risk-sensitive decision rule combines expected misdiagnosis cost, epistemic uncertainty, and claim-support thresholds to recommend a diagnosis, request further evidence, or defer. The contribution is a connected methodological account of evidence weighting, claim attribution, uncertainty decomposition, calibration assessment, and deferral logic tailored to clinical literature interpretation. The framework does not assume that semantic similarity equals clinical validity, nor that verbal model confidence is sufficient for safety-critical use. It specifies where retrieval error, contradictory evidence, model overconfidence, and cost misspecification enter the decision pathway, and it proposes evaluation logic based on calibration curves, decision-curve analysis, and claim-level evidence auditing. The approach provides a technically defensible basis for evidence-calibrated biomedical decision support prior to clinical validation.

References

1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (Vol. 33, pp. 9459–9474).

2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008).

3. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics.

4. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020). BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4), 1234–1240.

5. Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Arcas, B. A., ... Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172–180.

6. Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large language models in medicine. Nature Medicine, 29(8), 1930–1940.

7. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38.

8. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR.

9. Gal, Y., & Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on Machine Learning (Vol. 48, pp. 1050–1059). PMLR.

10. Zhang, R. (2025, December). Uncertainty-Aware RAG: Trustworthy Generation and Risk Control. In 2025 5th International Symposium on Artificial Intelligence and Big Data (AIBDF) (pp. 526-531). IEEE.

11. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (Vol. 30, pp. 6402–6413).

12. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems (Vol. 30, pp. 5574–5584).

13. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM.

14. Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6), 887.

15. Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008). GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ, 336(7650), 924–926.

16. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., ... Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71.

17. Sox, H. C., Higgins, M. C., & Owens, D. K. (2013). Medical decision making (2nd ed.). Wiley-Blackwell.

18. Elkan, C. (2001). The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence (pp. 973–978). Morgan Kaufmann.

19. Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., ... Kaplan, J. (2022). Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.

20. Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334.

Downloads

Published

2026-10-09

How to Cite

Bahuraksa Adriansyah, Milosz Galos, Aadi Bhandari, & Eshana Sibal. (2026). Evidence-Calibrated Biomedical Decision Support Using Retrieval-Augmented Large Language Models for Clinical Literature Interpretation and Uncertainty-Aware Diagnosis Prediction. International Journal of Clinical and Translational Medicine, 2(2). Retrieved from https://ijctmed.org/index.php/home/article/view/238