Abstract
Hyperspectral (HS) images are used in many fields to improve the analysis and understanding performance of captured scenes, as they contain a wide range of spectral information. However, the spatial resolution of hyperspectral images is usually very low, which limits their wide applicability in real tasks. To address the problem of low spatial resolution, super-resolution (SR) methods for hyperspectral images (HSI) have attracted widespread interest, which aims to mathematically generate high spatial resolution hyperspectral (HR-HS) images by combining degraded observational data: low spatial resolution hyperspectral (LR-HS) images and high resolution multispectral or RGB (HR-MS/RGB) images. Recently, paradigms based on deep learning have been widely explored as an alternative to automatically learn the inherent priors for the latent HR-HS images. These learning-based approaches are usually implemented in a fully supervised manner and require large external datasets including degraded observational data: LR-HS/HR-RGB images and corresponding HR-HS data, which are difficult to collect, especially for HSI SR scenarios. Therefore, in this study, a new unsupervised HSI SR method is proposed that uses only the observed LR-HS and HR-RGB images without any other external samples. Specifically, we use an RGB-driven deep generative network to learn the desired HR-HS images using a encoding-decoding-based network architecture. Since the observed HR-RGB images have a more detailed spatial structure and may be more suitable for two-dimensional convolution operations, we employ the observed HR-RGB images as input to the network as a conditional guide and adopt the observed LR-HS/HR-RGB images to formulate the loss function that guides the network learning. Experimental results on two HS image datasets show that our proposed unsupervised approach provides superior results compared to the SoTA deep learning paradigms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Akhtar, N., Shafait, F., Mian, A.: Sparse spatio-spectral representation for hyperspectral image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 63–78. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10584-0_5
Bach, S.H., He, B., Ratner, A., Ré, C.: Learning the structure of generative models without labeled data. In: International Conference on Machine Learning, pp. 273–282. PMLR (2017)
Bioucas-Dias, J.M., Plaza, A., Camps-Valls, G., Scheunders, P., Nasrabadi, N., Chanussot, J.: Hyperspectral remote sensing data analysis and future challenges. IEEE Geosci. Remote Sens. Mag. 1(2), 6–36 (2013)
Dian, R., Fang, L., Li, S.: Hyperspectral image super-resolution via non-local sparse tensor factorization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5344–5353 (2017)
Dian, R., Li, S., Guo, A., Fang, L.: Deep hyperspectral image sharpening. IEEE Trans. Neural Netw. Learn. Syst. 99, 1–11 (2018)
Dong, W., et al.: Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Trans. Image Process. 25(5), 2337–2352 (2016)
Fu, Y., Zhang, T., Zheng, Y., Zhang, D., Huang, H.: Hyperspectral image super-resolution with optimized RGB guidance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11661–11670 (2019)
Han, X.H., Chen, Y.W.: Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pp. 266–270. IEEE (2019)
Han, X.H., Shi, B., Zheng, Y.: Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Trans. Image Process. 27(11), 5625–5637 (2018)
Han, X.H., Shi, B., Zheng, Y.: SSF-CNN: spatial and spectral fusion with CNN for hyperspectral image super-resolution. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 2506–2510. IEEE (2018)
Han, X.H., Sun, Y., Chen, Y.W.: Residual component estimating CNN for image super-resolution. In: 2019 IEEE Fifth International Conference on Multimedia Big Data (BigMM), pp. 443–447. IEEE (2019)
Han, X.H., Zheng, Y., Chen, Y.W.: Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2019)
He, W., Zhang, H., Zhang, L., Shen, H.: Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration. IEEE Trans. Geosci. Remote Sens. 54(1), 178–188 (2015)
Huang, Q., Li, W., Hu, T., Tao, R.: Hyperspectral image super-resolution using generative adversarial network and residual learning. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3012–3016. IEEE (2019)
Kawakami, R., Matsushita, Y., Wright, J., Ben-Ezra, M., Tai, Y.W., Ikeuchi, K.: High-resolution hyperspectral imaging via matrix factorization. In: CVPR 2011, pp. 2329–2336. IEEE (2011)
Lanaras, C., Baltsavias, E., Schindler, K.: Hyperspectral super-resolution by coupled spectral unmixing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3586–3594 (2015)
Liu, Z., Zheng, Y., Han, X.H.: Unsupervised multispectral and hyperspectral image fusion with deep spatial and spectral priors. In: Proceedings of the Asian Conference on Computer Vision (2020)
Liu, Z., Zheng, Y., Han, X.H.: Deep unsupervised fusion learning for hyperspectral image super resolution. Sensors 21(7), 2348 (2021)
Lu, G., Fei, B.: Medical hyperspectral imaging: a review. J. Biomed. Opt. 19(1), 010901 (2014)
Mertens, S., et al.: Proximal hyperspectral imaging detects diurnal and drought-induced changes in maize physiology. Front. Plant Sci. 12, 240 (2021)
Park, S.M., Kim, Y.L.: Spectral super-resolution spectroscopy for biomedical applications. In: Advanced Chemical Microscopy for Life Science and Translational Medicine 2021, vol. 11656, p. 116560N. International Society for Optics and Photonics (2021)
Qu, Y., Qi, H., Kwan, C.: Unsupervised sparse dirichlet-net for hyperspectral image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2511–2520 (2018)
Sakthivel, S.P., Sivalingam, J.V., Shanmugam, S., et al.: Super-resolution mapping of hyperspectral images for estimating the water-spread area of peechi reservoir, southern india. J. Appl. Remote Sens. 8(1), 083510 (2014)
Saralıoğlu, E., Görmüş, E.T., Güngör, O.: Mineral exploration with hyperspectral image fusion. In: 2016 24th Signal Processing and Communication Application Conference (SIU), pp. 1281–1284. IEEE (2016)
Uezato, T., Hong, D., Yokoya, N., He, W.: Guided deep decoder: unsupervised image pair fusion. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12351, pp. 87–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58539-6_6
Wang, J., Kwon, S., Shim, B.: Generalized orthogonal matching pursuit. IEEE Trans. Signal Process. 60(12), 6202–6216 (2012)
Wycoff, E., Chan, T.H., Jia, K., Ma, W.K., Ma, Y.: A non-negative sparse promoting algorithm for high resolution hyperspectral imaging. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1409–1413. IEEE (2013)
Xie, Q., Zhou, M., Zhao, Q., Meng, D., Zuo, W., Xu, Z.: Multispectral and hyperspectral image fusion by MS/HS fusion net. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1585–1594 (2019)
Xu, J.L., Riccioli, C., Sun, D.W.: Comparison of hyperspectral imaging and computer vision for automatic differentiation of organically and conventionally farmed salmon. J. Food Eng. 196, 170–182 (2017)
Yokoya, N., Chan, J.C.W., Segl, K.: Potential of resolution-enhanced hyperspectral data for mineral mapping using simulated EnMAP and sentinel-2 images. Remote Sens. 8(3), 172 (2016)
Yokoya, N., Zhu, X.X., Plaza, A.: Multisensor coupled spectral unmixing for time-series analysis. IEEE Trans. Geosci. Remote Sens. 55(5), 2842–2857 (2017)
Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Signal Process. 128, 389–408 (2016)
Zhang, S., Liang, G., Pan, S., Zheng, L.: A fast medical image super resolution method based on deep learning network. IEEE Access 7, 12319–12327 (2018)
Zhao, Y., Yang, J., Zhang, Q., Song, L., Cheng, Y., Pan, Q.: Hyperspectral imagery super-resolution by sparse representation and spectral regularization. EURASIP J. Adv. Signal Process. 2011(1), 1–10 (2011)
Zhu, Z., Hou, J., Chen, J., Zeng, H., Zhou, J.: Residual component estimating CNN for image super-resolution, vol. 30, pp. 1423–1428 (2020)
Acknowledgement
This research was supported in part by the Grant-in Aid for Scientific Research from the Japanese Ministry for Education, Science, Culture and Sports (MEXT) under the Grant No. 20K11867, and JSPS KAKENHI Grant Number JP12345678.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Z., Han, XH. (2023). Deep RGB-Driven Learning Network for Unsupervised Hyperspectral Image Super-Resolution. In: Zheng, Y., Keleş, H.Y., Koniusz, P. (eds) Computer Vision – ACCV 2022 Workshops. ACCV 2022. Lecture Notes in Computer Science, vol 13848. Springer, Cham. https://doi.org/10.1007/978-3-031-27066-6_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-27066-6_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27065-9
Online ISBN: 978-3-031-27066-6
eBook Packages: Computer ScienceComputer Science (R0)