An Improved Pan-Sharpening Method Based on Convolutional Autoencoder and Guided Filter (CAE-GF) on Quickbird Images

Authors

DOI:

https://doi.org/10.11137/1982-3908_2024_47_58601

Keywords:

Deep Learning, Image Fusion, Remote Sensing

Abstract

Pan-sharpening is the process of combining high-resolution panchromatic and low-resolution multispectral images to generate a high-resolution multispectral image. However, some pan-sharpening methods have poor preservation of spectral and spatial information. In this context, the aim of this study is to improve the pan-sharpening method based on a convolutional autoencoder and a guided filter (CAE-GF), by developing a new Convolutional Autoencoder (CAE) network. A CAE network was designed and trained to generate original panchromatic images from their spatially degraded versions. The trained network was used to improve the spatial details of the intensity component of the multispectral image obtained through the adaptive intensity-hue-saturation (AIHS) method. The pan-sharpening process is achieved by applying the multi-scale guided filter to improve the original PAN image using the enhanced intensity component and by injecting the detail map into the multispectral image. To analyze the preservation of the spectral and spatial information in the proposed method, full-reference and no-reference quality indices were calculated, and a visual analysis was performed. These analyses were compared with traditional pan-sharpening methods of component substitution. The results showed the developed method’s efficacy in generating Pan-sharpened QuickBird images.

Author Biography

Jessica da Silva Costa, Universidade Federal do Paraná, Programa de Pós-Graduação em Ciências Geodésicas

Universidade Federal do Paraná, Curitiba, PR, Brasil

References

Aiazzi, B., Alparone, L., Baronti, S., Garzelli, A. & Selva, M. 2006, 'MTF-tailored multiscale fusion of high-resolution MS and Pan imagery', Photogramm. Eng. Remote Sens., vol. 72, no. 5, pp. 591-6. https://doi.org/10.14358/PERS.72.5.591

Al Smadi, A., Yang, S., Abugabah, A. Alzubi, A. A. & Sanzogni, L. 2022, 'A Pansharpening Based on the Non-Subsampled Contourlet Transform and Convolutional Autoencoder: Application to QuickBird Imagery', IEEE Access, vol. 10, pp. 44778-88. https://doi.org/10.1109/ACCESS.2022.3169698

Al Smadi, A., Yang, S., Kai, Z., Mehmood, A., Wang, M. & Alsanabani, A. 2021, 'Pansharpening based on convolutional autoencoder and multi-scale guided filter', J Image Video Proc, vol. 25. https://doi.org/10.1186/s13640-021-00565-3

Alcaras, E., Parente, C. & Vallario, A. 2021, 'Automation of Pan-Sharpening Methods for Pléiades Images Using GIS Basic Functions', Remote Sens., vol. 13, no. 8, pp.1550. https://doi.org/10.3390/rs13081550

Alparone, L., Aiazzi, B., Baronti, S., Garzelli, A., Nencini, F. & Selva, M. 2008, 'Multispectral and panchromatic data fusion assessment without reference', Photogramm. Eng. Remote Sens, vol. 74, no. 2, pp.193–200. https://doi.org/10.14358/PERS.74.2.193

Azarang, A, Manoochehri, H. E. & Kehtarnavaz, N. 2019, 'Convolutional Autoencoder-Based Multispectral Image Fusion', IEEE Access, vol. 7, pp. 35673-83. https://doi.org/10.1109/ACCESS.2019.2905511

Chavez, A. & Kwarteng, P. 1989, 'Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis', Photogramm. Eng. Remote Sens., vol. 55, no. 1, pp. 339-48.

Dadrass Javan, F., Samadzadegan, F., Mehravar, S., Toosi, A., Khatami, R. & Stein, A. 2021, 'A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery', ISPRS Journal of Photogrammetry and Remote Sensing, vol.171, pp. 101-17. https://doi.org/10.1016/j.isprsjprs.2020.11.001

Esri 2023. Fundamentals of Panchromatic Sharpening. ArcGIS for Desktop, viewed 27 March 2023, <https://desktop.arcgis.com/en/arcmap/10.3/manage-data/raster-and-images/fundamentals-of-panchromatic-sharpening.htm>.

Goodfellow, I., Bengio, Y. & Courville, A. 2016, Deep Learning, MITPress.

Google Colab. 2023, Google Colab, viewed 27 March 2023, <https://colab.research.google.com/notebooks/intro.ipynb>.

Ioffe, S. & Szegedy, C. 2015, 'Batch normalization: Accelerating deep network training by reducing internal covariate shift', International conference on machine learning, Lille, France, vol. 37, p. 448-56.

https://doi.org/10.48550/arXiv.1502.03167

Jiang, D., Zhuang, D., Huang, Y. & Fu, J. 2011, 'Survey of multispectral image fusion techniques in remote sensing applications', Image fusion its Appl., pp.1–23. https://doi.org/10.5772/10548

Kaur, G., Saini, K.S., Singh, D. & Kaur, M. 2021, 'A Comprehensive Study on Computational Pansharpening Techniques for Remote Sensing Images', Archives of Computational Methods in Engineering, vol. 28, pp. 4961–978. https://doi.org/10.1007/s11831-021-09565-y

Klonus, S. & Ehlers, M. 2009, 'Performance of evaluation methods in image fusion', 12th International Conference on Information Fusion, IEEE, Seattle, WA, pp. 1409-16.

Lee J., Seo S. & Kim, M. 2021, 'SIPSA-Net: Shift-Invariant Pan Sharpening with Moving Object Alignment for Satellite Imagery', 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, pp. 10161-69.

https://doi.org/10.1109/CVPR46437.2021.01003

Mahyari, A.G. & Yazdi. M. 2011, 'Panchromatic and multispectral image fusion based on maximization of both spectral and spatial similarities', IEEE Tran. Geosci. Remote Sens. vol. 49, no. 6, pp. 1976-85. https://doi.org/10.1109/TGRS.2010.2103944

Meng, X., Shen, H., S., Li, H., Zhang, L. & Randi Fu, R. 2019, 'Review of the Pansharpening Methods for Remote Sensing Images Based on the Idea of Meta-analysis: Practical Discussion and Challenges', Information Fusion, vol. 46, pp. 102-13. https://doi.org/10.1016/j.inffus.2018.05.006

Nair, V. & Hinton, G. E. 2010, 'Rectified Linear Units Improve Restricted Boltzmann Machines', International Conference on Machine Learning, Haifa, Israel, pp. 807-14.

Ochotorena, C. N. & Yamashita, Y. 2020, 'Anisotropic guided filtering', IEEE Trans. Image Process., vol. 29, pp. 1397–412. https://doi.org/10.1109/TIP.2019.2941326.

Otazu, X., Gonzalez-Audícana, M., Fors, O. & Núnez, J. 2005, 'Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods', IEEE Trans. Geosci. Remote Sens. vol. 43, no. 10, pp. 2376-85. https://doi.org/10.1109/TGRS.2005.856106

Ouahab, A. & Belbachir, M. F. 2020, 'A comparison analysis of pan-sharpening methods on Alsat-2A images', 2020 2nd International Conference on Mathematics and Information Technology (ICMIT), Adrar, Algeria, pp. 138-41. https://doi.org/10.1109/ICMIT47780.2020.9046990

Palsson, F., Sveinsson, J. R., Ulfarsson, M. O. & Benediktsson, J. A. 2016, 'Quantitative quality evaluation of pansharpened imagery: consistency versus synthesis', IEEE Trans. Geosci. Remote Sens., vol. 54, no. 3, pp. 1247-59. https://doi.org/10.1109/TGRS.2015.2476513

Rahmani, S., Strait, M., Merkurjev, D., Moeller, M. & Wittman, T. 2010, 'An adaptive IHS pan-sharpening method', IEEE Geosci.Remote Sens. Lett., vol.7, no. 4, pp.746–50. https://doi.org/10.1109/LGRS.2010.2046715

Ranchin T. & Wald, L. 2000, 'Fusion of high spatial and spectral resolution images: The ARSIS concept and its implementation', Photogramm. Eng. Remote Sens., vol. 66, no. 1, pp. 49-61.

Rashmi, S., Addamani, S., Venkat, & Ravikiran, S. 2014, 'Spectal Angle Mapper Algorithm for remote Sensing Image Classification', International Journal of Innovative Science, Engineering & Technology, vol. 1, no. 4, pp. 201-5.

Tarchouli, M., Pelurson, S., Guionnet, T., Hamidouche, W., Outtas, M. & Deforges, O. 2022, 'Patch-based image coding with end-to-end learned codec using overlapping', Computer Science & Information Technology, vol. 12, no. 23, pp. 53-63. https://doi.org/10.5121/csit.2022.122305

Tu, T.-M., Su, S.-C., Shyu, H.-C. & Huang, P. S. 2001, 'A new look at IHS-like image fusion methods', Inf. Fusion, vol.2, no.3, pp.177-86. https://doi.org/10.1016/S1566-2535(01)00036-7

Vivone, G., Alparone, L., Chanussot, J., Dalla Mura, M., Garzelli, A., Licciardi, G. A., Restaino, R. & Wald, L. 2015, 'A Critical Comparison Among Pansharpening Algorithms', IEEE Transactions on Geoscience and Remote Sensing, vol. 53, no. 5, pp. 2565-86. https://doi.org/10.1109/TGRS.2014.2361734

Wald, L. 2002, Data Fusion: Definitions and Architectures Fusion of Images of Different Spatial Resolutions, Presses des MINES, Paris, France.

Wang, Z. & Bovik, A. C. 2002, 'A universal image quality index', IEEE Signal Process. Lett., vol. 9, no. 3, pp. 81–4. https://doi.org/10.1109/97.995823

Xu, B., Wang, N., Chen, T. & Li, M. 2015, 'Empirical Evaluation of Rectified Activations in Convolutional Network', ArXiv e-prints.

https://doi.org/10.48550/arXiv.1505.00853

Yang, Y., Tong, S., Huang, S. & Lin, P. 2015, 'Multifocus image fusion based on NSCT and focused area detection', IEEE Sensors J., vol. 15, no. 5, pp. 2824-38. https://doi.org/10.1109/JSEN.2014.2380153

Zhou, J., Civco, D.L. & Silander, J.A. 1998, 'A wavelet transform method to merge Landsat TM and SPOT panchromatic data', International Journal of Remote Sensing, vol. 19, no. 4, pp.743-57. https://doi.org/10.1080/014311698215973

Zhu, D., Cheng, X., Zhang, F. Yao, X., Gao, Y. & Liu, Y. 2020, 'Spatial interpolation using conditional generative adversarial neural networks', International Journal of Geographical Information Science, vol. 34, no. 4, pp.735-58. https://doi.org/10.1080/13658816.2019.1599122

Downloads

Published

2024-12-20

Issue

Vol. 47 (2024): Anuário do Instituto de Geociências

Section

Geography

License

Copyright (c) 2024 Anuário do Instituto de Geociências

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

This journal is licensed under a Creative Commons — Attribution 4.0 International — CC BY 4.0, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.