Enhancing Spectral Quality through Multisource Fusion of SAR and Optical Imagery Using Deep Learning
Article Sidebar
Main Article Content
Abstract
This study evaluates the effectiveness of a multi-source satellite image fusion method using deep learning to enhance spectral feature quality in urban environments. The input data consist of synthetic aperture radar (SAR) images from Sentinel 1 and optical images from Landsat 9. Two deep learning models were implemented: patch-wise shallow convolutional neural networks (CNN) and convolutional autoencoders (CAE). Each of Red-Green-Blue optical band was fused separately with the VV/VH ratio image derived from radar data. The fusion results were assessed using RMSE, SSIM, UIQI indices, and the Pearson correlation coefficient. The CAE demonstrated better spectral reconstruction capability with lower RMSE and higher SSIM across all three bands. Conversely, the CNN model achieved higher UIQI on some bands and produced images with visually superior sharpness. Visual assessment indicated that CNN better preserves edge details and fine structures, while CAE generated smoother images but with some blurring of objects. These results suggest that deep learning-based fusion methods hold great potential for improving input image quality for urban analysis in areas affected by cloud cover.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
References
M. Schmitt, L. H. Hughes and X. X. Zhu, “The SEN1-2 Dataset for Deep Learning in SAR-Optical Data Fusion,” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci., vol. IV-1, pp. 141–146, 2018.
Y. Ban, O. Yousif and H. Hu, “Fusion of SAR and Optical Data for Urban Land Cover Mapping and Change Detection,” Global Urban Monitoring and Assessment through Earth Observation, CRC Press, pp. 353-386, 2014.
C. Pohl and J. L. Van Genderen, “Multisensor Image Fusion in Remote Sensing: Concepts, Methods and Applications,” Int. J. Remote Sens., vol. 19, no. 5, pp. 823–854, 1998.
Y. Jiao and R. S. Blum, “A Fusion Method of SAR and Optical Images for Urban Object Extraction,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., vol. XXXVII, part B7, Beijing, pp. 1257-1260, 2008.
C. Hua, Z. Tang, K. Zhang, W. Jiang and H. Wang, “Study on Infrared Camouflage of Landing Craft and Camouflage Effect Evaluation,” Infrared Technol., vol. 30, pp. 379–383, 2008.
M. N. Do and M. Vetterli, “The Contourlet Transform: An Efficient Directional Multiresolution Image Representation,” IEEE Trans. Image Process., vol. 14, pp. 2091–2106, 2005.
Q. Xiao-Bo, Y. Jing-Wen, X. Hong-Zhi and Z. Zi-Qian, “Image Fusion Algorithm Based on Spatial Frequency-Motivated Pulse Coupled Neural Networks in Nonsubsampled Contourlet Transform Domain,” Acta Autom. Sin., vol. 34, pp. 1508–1514, 2008
A. L. Da Cunha, J. P. Zhou and M. N. Do, “The Nonsubsampled Contourlet Transform: Theory, Design, and Applications,” IEEE Trans. Image Process., vol. 15, pp. 3089–3101, 2006.
D. Limpitlaw and R. Gens, “Dambo Mapping for Environmental Monitoring Using Landsat TM and SAR Imagery: Case Study In The Zambian Copperbelt,” Int. J. Remote Sens., vol. 27, pp. 4839–4845, 2006.
L. Zhang, L. Zhang and B. Du, “Deep Learning for Remote Sensing Data,” IEEE Geosci. Remote Sens. Mag., vol. 4, no. 2, pp. 22–40, 2016.
M. Schmitt and X. Zhu, “Data Fusion and Remote Sensing - An Ever-Growing Relationship,” IEEE Geosci. Remote Sens. Mag., vol. 4, no. 4, pp. 6–23, 2016.
H. C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura and R. M. Summers, “Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning,” IEEE Trans. Med. Imaging, vol. 35, no. 5, pp. 1285–1298, 2016.
S. Hochreiter and J. Schmidhuber, “Long Short-term Memory,” Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
Y. Zhang, F. Liu and H. Wang, “Dual-branch Convolutional Networks for SAR and Multispectral Data Fusion,” Remote Sens. Lett., vol. 11, no. 4, pp. 345–357, 2020.
L. Xiong, Q. Zhao and Y. Chen, “SAR-to-Optical Image Translation Using Conditional GANs,” IEEE Trans. Geosci. Remote Sens., vol. 61, pp. 1–10, 2023.
M. Liu, J. Zhou and Y. Tang, “Sentinel-1 and Sentinel-2 Data Fusion using U-Net for Land Cover Classification,” ISPRS J. Photogramm. Remote Sens., vol. 187, pp. 113–124, 2022.
B. Li and Z. Wang, “Urban Structure Enhancement using Residual Learning on SAR-Optical Fused Imagery,” Comput. Environ. Urban Syst., vol. 76, pp. 93–102, 2019.
R. Ding, X. Hu and C. Wang, “Benchmark Study of Fusion Methods for Multisensor Imagery: IHS, PCA, Wavelet, and HIS+NSCT,” Remote Sens., vol. 17, no. 2, pp. 225–240, 2025.
S. K. Roy, A. Deria, D. Hong, B. Rasti, A. Plaza and J. Chanussot, “Multimodal Fusion Transformer for Remote Sensing Image Classification,” IEEE Trans. Geosci. and Remote Sens., vol. 61, pp. 1-20, 2023.
Statistical Yearbook of Vietnam 2023, General Statistics Office, Hanoi: Statistical Publishing House, 2024.
K. Lewinska, S. Ernst, D. Frantz, U. Leser and P. Hostert, “Global Overview of Usable Landsat and Sentinel-2 Data for 1982–2023,” Data in Brief, vol. 57, pp. 111054, 2024.
A. Doerry, “Introduction to Synthetic Aperture Radar,” in 2019 IEEE Radar Conf., Boston, MA, USA, pp. 1–90, 2019.
R. Torres, P. Snoeij, D. Geudtner, et al., “GMES Sentinel-1 Mission,” Remote Sens. Environ., vol. 120, pp. 9–24, 2012.
Y. Zhang, Y. Ban and Y. Hu, “Fusion of Sentinel-1 and Sentinel-2 Time Series for Urban Land Cover Classification using A Dual-Branch CNN,” ISPRS J. Photogramm. Remote Sens., vol. 164, pp. 230–243, 2020.
Y. Zhang, J. Liu and D. Li, “Remote Sensing Image Fusion using Deep Learning: A Review,” Inf. Fusion, vol. 52, pp. 20–34, 2019.
Y. Liu, L. Wang and M. Xu, “SAR and Optical Image Fusion via Deep Residual Networks for Urban Mapping,” ISPRS J. Photogramm. Remote Sens., vol. 178, pp. 110–124, 2021.
Z. Cheng, H. Sun, M. Takeuchi and J. Katto, “Deep Convolutional AutoEncoder-based Lossy Image Compression,” in Proc. the 2018 Pic. Cod. Symp., San Francisco, CA, USA, pp. 253-257, 2018.
V. Turchenko, E. Chalmers and A. Luczak, “A Deep Convolutional Auto-Encoder with Pooling–Unpooling Layers in Caffe,” Int. J. Comput., vol. 18, no. 1, pp. 8–31, 2019.
Y. Xu, K. Fu and L. Zhang, "A Multi-scale Feature Fusion Network for Cloud Removal in Remote Sensing Images," ISPRS J. Photogramm. Remote Sens., vol. 161, pp. 182–194, 2020.
Z. Alzamili, K. Danach and M. Frikha, “Deep Learning-Based Patch-Wise Illumination Estimation for Enhanced Multi-Exposure Fusion,” IEEE Access, vol. 11, 120642
- 120652, 2023.
Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, "Image Quality Assessment: From Error Visibility to Structural Similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004.
K. Guo, X. Li, H. Zang and T. Fan, “Multi-Modal Medical Image Fusion Based on FusionNet in YIQ Color Space,” Entropy, vol. 22, no. 12, pp. 1423, 2020.
Z. Wang and A. C. Bovik, "A Universal Image Quality Index," IEEE Signal Process. Lett., vol. 9, no. 3, pp. 81–84, 2002.
C. E. Shannon, “A Mathematical Theory of Communication,” Bell Syst. Techn. J., vol. XXVII, no. 3, pp. 379-423, 1948.
R. C. Gonzalez and R. E. Woods, Digital Image Processing, 4th Edn., Pearson, New York, 2018.
A. M. Eskicioglu and P. S, Fisher, “Image Quality Measures and Their Performance,” IEEE Trans. Commun., vol. 43, no. 12, pp. 2959–2965, 1995.
C. Xydeas and V. Petrovic, “Objective Image Fusion Performance Measure,” Electron. Lett., vol. 36, no. 4, pp. 308–309, 2000.
Y. Luo, J. Zhang and H. Li, “DAFCNN: Dual-Attention Fusion Convolutional Neural Network for SAR And Multispectral Image Fusion,” Remote Sens. Lett., vol. 14, no. 5, pp. 435–450, 2023.
Y. Yuhendra and M. Minarni, “Optical SAR Images Fusion: Comparative Analysis of Resulting Images Data,” MATEC Web of Conf., vol. 215, pp. 01002, 2018.