Geospatial Deep Learning for Predicting Public Health Risks Through Multimodal Analysis of Social Media Sentiment and Urban Environmental Imagery
Keywords:
multimodal deep learning; geospatial public health; social media sentiment; street-level imagery; risk calibration; graph neural networksAbstract
Public health risk varies sharply across small urban areas, but conventional surveillance is often too sparse, delayed, or aggregated to support neighborhood-scale screening. Social media sentiment and street-level imagery provide complementary observations: text streams capture expressed distress, lifestyle change, and service complaints, while environmental images encode visible greenness, built density, disorder, and exposure conditions. This manuscript develops a geospatial deep learning framework that fuses geotagged social media sentiment with urban environmental imagery and spatial covariates to estimate area-time public health risk. The contribution is a complete methodological architecture rather than a reported field trial. The framework specifies reliability-weighted sentiment aggregation, pretrained image encoding, gated multimodal fusion, graph-based spatial regularization, calibrated probabilistic risk estimation, and uncertainty-aware decision thresholds. It treats spatial support mismatch, geotag imprecision, platform selection bias, temporal misalignment between posts and images, and visually confounded urban form as first-class modeling problems. The central argument is that multimodal gains depend less on adding modalities than on representing when each modality is trustworthy at a given location and time. Spatial blocking, calibration, subgroup assessment, and decision-cost analysis are therefore integrated into the evaluation logic. The resulting framework supports public health screening and urban planning as a triage aid, not as a substitute for clinical or epidemiological diagnosis. It also provides an auditable basis for assessing where social media signals enrich environmental imagery and where imagery corrects sentiment-only blind spots.
References
1. Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of Google Flu: Traps in big data analysis. Science, 343(6176), 1203–1205.
2. Paul, M. J., & Dredze, M. (2011). You are what you tweet: Analyzing Twitter for public health. Proceedings of the International AAAI Conference on Web and Social Media, 5(1), 265–272.
3. Nguyen, Q. C., Li, D., Meng, H.-W., Kath, S., Nsoesie, E., Li, F., & Wen, M. (2016). Geotagged US tweets as predictors of county-level health outcomes, 2015. American Journal of Public Health, 106(12), 2241–2247.
4. Goodchild, M. F. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69(4), 211–221.
5. Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E. L., & Fei-Fei, L. (2017). Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences, 114(50), 13108–13113.
6. Aman, J., Matisziw, T. C., Kim, J. B., & Luo, D. (2022, April). Sensing the city: Leveraging geotagged social media posts and street view imagery to model urban streetscapes using deep neural networks. In Proceedings of the International Conference on Computer-Aided Architectural Design Research in Asia (pp. 595-604). Association for Computer-Aided Architectural Design Research in Asia.
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
8. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 770–778).
9. Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations.
10. Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443.
11. De Choudhury, M., Gamon, M., Counts, S., & Horvitz, E. (2013). Predicting depression via social media. Proceedings of the International AAAI Conference on Web and Social Media, 7(1), 128–137.
12. Hartig, T., Mitchell, R., de Vries, S., & Frumkin, H. (2014). Nature and health. Annual Review of Public Health, 35, 207–228.
13. Anselin, L. (1995). Local indicators of spatial association—LISA. Geographical Analysis, 27(2), 93–115.
14. Kendall, A., & Gal, Y. (2017). What uncertainties do we need in Bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 5574–5584.
15. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (pp. 1321–1330).
16. Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., & Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929.
17. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 International Journal of Clinical and Translational Medicine

This work is licensed under a Creative Commons Attribution 4.0 International License.
This article is published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.



