Multimodal Deep Learning for Joint Estimation of Heart Rate, Respiratory Rate, and Activity States
Keywords:
multimodal deep learning; physiological monitoring; heart rate; respiratory rate; activity recognition; wearable systems; algorithmic fairness; edge deploymentAbstract
Multimodal deep learning is transforming physiological monitoring by integrating signals from electrocardiography, photoplethysmography, accelerometry, and contextual sensing to jointly estimate heart rate, respiratory rate, and activity states. Single-sensor approaches remain vulnerable to motion artifacts, sensor displacement, and physiological confounding, particularly during ambulatory use. This paper presents a system-level analysis of multimodal deep learning architectures for joint physiological and behavioral inference. We examine early, late, and hybrid fusion strategies, recurrent and attention-based temporal modeling, and their trade-offs in accuracy, latency, energy consumption, and interpretability. The discussion extends beyond model architecture to data acquisition, preprocessing infrastructure, annotation governance, bias management, and deployment across wearable and clinical edge computing platforms. We analyze robustness under distribution shift, fairness across demographic and clinical subpopulations, and regulatory expectations for continuous monitoring systems. A comparative perspective across consumer wearables, hospital telemetry, and remote patient monitoring illustrates structural differences in reliability, privacy, and sustainment. The paper argues that joint estimation should be treated as a socio-technical infrastructure problem, not solely as a pattern recognition task, and offers governance-oriented design principles for developing equitable, robust, and operationally sustainable multimodal monitoring systems.
References
1. Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443. https://doi.org/10.1109/TPAMI.2018.2798607
2. Zhang, Z., Pi, Z., & Liu, B. (2015). TROIKA: A general framework for heart rate monitoring using wrist-type photoplethysmographic signals during intensive physical exercise. IEEE Transactions on Biomedical Engineering, 62(2), 522–531. https://doi.org/10.1109/TBME.2014.2359372
3. Temko, A. (2017). Accurate heart rate monitoring during physical exercises using PPG. IEEE Transactions on Biomedical Engineering, 64(9), 2016–2024. https://doi.org/10.1109/TBME.2017.2676243
4. Reiss, A., Indlekofer, I., Schmidt, P., & Van Laerhoven, K. (2019). Deep PPG: Large-scale heart rate estimation with convolutional neural networks. Sensors, 19(14), 3079. https://doi.org/10.3390/s19143079
5. Biswas, D., Everson, L., Liu, M., Panwar, M., Verhoef, B. E., Patki, S., Kim, C. H., Acharyya, A., Van Hoof, C., Konijnenburg, M., & Van Helleputte, N. (2019). CorNET: Deep learning framework for PPG-based heart rate estimation and biometric identification in ambulant environment. IEEE Transactions on Biomedical Circuits and Systems, 13(2), 282–291. https://doi.org/10.1109/TBCAS.2019.2892297
6. Charlton, P. H., Birrenkott, D. A., Bonnici, T., Pimentel, M. A. F., Johnson, A. E. W., Alastruey, J., Tarassenko, L., Watkinson, P. J., Beale, R., & Clifton, D. A. (2018). Breathing rate estimation from the electrocardiogram and photoplethysmogram: A review. IEEE Reviews in Biomedical Engineering, 11, 2–20. https://doi.org/10.1109/RBME.2017.2763681
7. Ordóñez, F. J., & Roggen, D. (2016). Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors, 16(1), 115. https://doi.org/10.3390/s16010115
8. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017).Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998–6008).
9. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
10. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.
11. Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations. arXiv:1412.6980
12. Hannun, A. Y., Rajpurkar, P., Haghpanahi, M., Tison, G. H., Bourn, C., Turakhia, M. P., & Ng, A. Y. (2019). Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nature Medicine, 25(1), 65–69.
13. Karlen, W., Raman, S., Ansermino, J. M., & Dumont, G. A. (2013). Multiparameter respiratory rate estimation from the photoplethysmogram. IEEE Transactions on Biomedical Engineering, 60(7), 1946–1953.
14. Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018). Multimodal deep learning for activity and context recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(4), 157.
15. Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86–92.
16. Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020). Racial bias in pulse oximetry measurement. New England Journal of Medicine, 383(25), 2477–2478.
17. Mittelstadt, B. D. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507.
18. Zheng, X., Hu, S., Dwyer, V., Barrett, L., & Derakhshani, M. (2026). Joint attention mechanism learning to facilitate opto-physiological monitoring during physical activity. Biomedical Signal Processing and Control, 113, 108949.
19. Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019). What clinicians want: Contextualizing explainable machine learning for clinical end use. arXiv preprint arXiv:1905.05134.
20. Voigt, P., & Von dem Bussche, A. (2017). The EU General Data Protection Regulation (GDPR): A practical guide. Springer.
21. World Health Organization. (2021). Global strategy on digital health 2020–2025. World Health Organization.
22. Allen, J. (2007). Photoplethysmography and its application in clinical physiological measurement. Physiological Measurement, 28(3), R1–R39.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 International Journal of Clinical and Translational Medicine
This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
