Gustau Camps-Valls is Professor of Electrical Engineering and Lead Researcher in the Image Processing Laboratory (IPL) at the Universitat de València. His interests include development of statistical learning, mainly kernel machines and neural networks, for Earth sciences, from remote sensing to geoscience data analysis. Models efficiency and accuracy but also interpretability, consistency and causal discovery are driving his agenda on AI for Earth and climate.

Devis Tuia, PhD, is Associate Professor at the Ecole Polytechnique Fédérale de Lausanne (EPFL). He leads the Environmental Computational Science and Earth Observation laboratory, which focuses on the processing of Earth observation data with computational methods to advance Environmental science.

Xiao Xiang Zhu is Professor of Data Science in Earth Observation and Director of the Munich AI Future Lab AI4EO at the Technical University of Munich and heads the Department EO Data Science at the German Aerospace Center. Her lab develops innovative machine learning methods and big data analytics solutions to extract large scale geo-information from big Earth observation data, aiming at tackling societal grand challenges, e.g. Urbanization, UN's SDGs and Climate Change.

Markus Reichstein is Director of the Biogeochemical Integration Department at the Max-Planck- Institute for Biogeochemistry and Professor for Global Geoecology at the University of Jena. His main research interests include the response and feedback of ecosystems (vegetation and soils) to climatic variability with an Earth system perspective, considering coupled carbon, water and nutrient cycles. He has been tackling these topics with applied statistical learning for more than 15 years.

Foreword xvi
by Vipin Kumar, Regents Professor, University of Minnesota Acknowledgments xvii List of Contributors xviii List of Acronyms xxiv 1 Introduction 1Gustau Camps-Valls, Xiao Xiang Zhu, Devis Tuia, and Markus Reichstein 1.1 A Taxonomy of Deep Learning Approaches 2 1.2 Deep Learning in Remote Sensing 3 1.3 Deep Learning in Geosciences and Climate 7 1.4 Book Structure and Roadmap 9 Part I Deep Learning to Extract Information from Remote Sensing Images 13 2 Learning Unsupervised Feature Representations of Remote Sensing Data with Sparse Convolutional Networks 15Jose E. Adsuara, Manuel Campos-Taberner, Javier García-Haro, Carlo Gatta, Adriana Romero, and Gustau Camps-Valls 2.1 Introduction 15 2.2 Sparse Unsupervised Convolutional Networks 17 2.2.1 Sparsity as the Guiding Criterion 17 2.2.2 The EPLS Algorithm 18 2.2.3 Remarks 18 2.3 Applications 19 2.3.1 Hyperspectral Image Classification 19 2.3.2 Multisensor Image Fusion 21 2.4 Conclusions 22 3 Generative Adversarial Networks in the Geosciences 24Gonzalo Mateo-García, Valero Laparra, Christian Requena-Mesa, and Luis Gómez-Chova 3.1 Introduction 24 3.2 Generative Adversarial Networks 25 3.2.1 Unsupervised GANs 25 3.2.2 Conditional GANs 26 3.2.3 Cycle-consistent GANs 27 3.3 GANs in Remote Sensing and Geosciences 28 3.3.1 GANs in Earth Observation 28 3.3.2 Conditional GANs in Earth Observation 30 3.3.3 CycleGANs in Earth Observation 30 3.4 Applications of GANs in Earth Observation 31 3.4.1 Domain Adaptation Across Satellites 31 3.4.2 Learning to Emulate Earth Systems from Observations 33 3.5 Conclusions and Perspectives 36 4 Deep Self-taught Learning in Remote Sensing 37Ribana Roscher 4.1 Introduction 37 4.2 Sparse Representation 38 4.2.1 Dictionary Learning 39 4.2.2 Self-taught Learning 40 4.3 Deep Self-taught Learning 40 4.3.1 Application Example 43 4.3.2 Relation to Deep Neural Networks 44 4.4 Conclusion 45 5 Deep Learning-based Semantic Segmentation in Remote Sensing 46Devis Tuia, Diego Marcos, Konrad Schindler, and Bertrand Le Saux 5.1 Introduction 46 5.2 Literature Review 47 5.3 Basics on Deep Semantic Segmentation: Computer Vision Models 49 5.3.1 Architectures for Image Data 49 5.3.2 Architectures for Point-clouds 52 5.4 Selected Examples 55 5.4.1 Encoding Invariances to Train Smaller Models: The example of Rotation 55 5.4.2 Processing 3D Point Clouds as a Bundle of Images: SnapNet 59 5.4.3 Lake Ice Detection from Earth and from Space 62 5.5 Concluding Remarks 66 6 Object Detection in Remote Sensing 67Jian Ding, Jinwang Wang, Wen Yang, and Gui-Song Xia 6.1 Introduction 67 6.1.1 Problem Description 67 6.1.2 Problem Settings of Object Detection 69 6.1.3 Object Representation in Remote Sensing 69 6.1.4 Evaluation Metrics 69 6.1.4.1 Precision-Recall Curve 70 6.1.4.2 Average Precision and Mean Average Precision 71 6.1.5 Applications 71 6.2 Preliminaries on Object Detection with Deep Models 72 6.2.1 Two-stage Algorithms 72 6.2.1.1 R-CNNs 72 6.2.1.2 R-fcn 73 6.2.2 One-stage Algorithms 73 6.2.2.1 Yolo 73 6.2.2.2 Ssd 73 6.3 Object Detection in Optical RS Images 75 6.3.1 Related Works 75 6.3.1.1 Scale Variance 75 6.3.1.2 Orientation Variance 75 6.3.1.3 Oriented Object Detection 75 6.3.1.4 Detecting in Large-size Images 76 6.3.2 Datasets and Benchmark 77 6.3.2.1 Dota 77 6.3.2.2 VisDrone 77 6.3.2.3 Dior 77 6.3.2.4 xView 77 6.3.3 Two Representative Object Detectors in Optical RS Images 78 6.3.3.1 Mask OBB 78 6.3.3.2 RoI Transformer 82 6.4 Object Detection in SAR Images 86 6.4.1 Challenges of Detection in SAR Images 86 6.4.2 Related Works 86 6.4.3 Datasets and Benchmarks 88 6.5 Conclusion 89 7 Deep Domain Adaptation in Earth Observation 90Benjamin Kellenberger, Onur Tasar, Bharath Bhushan Damodaran, Nicolas Courty, and Devis Tuia 7.1 Introduction 90 7.2 Families of Methodologies 91 7.3 Selected Examples 93 7.3.1 Adapting the Inner Representation 93 7.3.2 Adapting the Inputs Distribution 97 7.3.3 Using (few, well chosen) Labels from the Target Domain 100 7.4 Concluding remarks 104 8 Recurrent Neural Networks and the Temporal Component 105Marco Körner and Marc Rußwurm 8.1 Recurrent Neural Networks 106 8.1.1 Training RNNs 107 8.1.1.1 Exploding and Vanishing Gradients 107 8.1.1.2 Circumventing Exploding and Vanishing Gradients 109 8.2 Gated Variants of RNNs 111 8.2.1 Long Short-term Memory Networks 111 8.2.1.1 The Cell State c t and the Hidden State h t 112 8.2.1.2 The Forget Gate f t 112 8.2.1.3 The Modulation Gate V T and the Input Gate I T 112 8.2.1.4 The Output Gate o t 112 8.2.1.5 Training LSTM Networks 113 8.2.2 Other Gated Variants 113 8.3 Representative Capabilities of Recurrent Networks 114 8.3.1 Recurrent Neural Network Topologies 114 8.3.2 Experiments 115 8.4 Application in Earth Sciences 117 8.5 Conclusion 118 9 Deep Learning for Image Matching and Co-registration 120Maria Vakalopoulou, Stergios Christodoulidis, Mihir Sahasrabudhe, and Nikos Paragios 9.1 Introduction 120 9.2 Literature Review 123 9.2.1 Classical Approaches 123 9.2.2 Deep Learning Techniques for Image Matching 124 9.2.3 Deep Learning Techniques for Image Registration 125 9.3 Image Registration with Deep Learning 126 9.3.1 2D Linear and Deformable Transformer 126 9.3.2 Network Architectures 127 9.3.3 Optimization Strategy 128 9.3.4 Dataset and Implementation Details 129 9.3.5 Experimental Results 129 9.4 Conclusion and Future Research 134 9.4.1 Challenges and Opportunities 134 9.4.1.1 Dataset with Annotations 134 9.4.1.2 Dimensionality of Data 135 9.4.1.3 Multitemporal Datasets 135 9.4.1.4 Robustness to Changed Areas 135 10 Multisource Remote Sensing Image Fusion 136Wei He, Danfeng Hong, Giuseppe Scarpa, Tatsumi Uezato, and Naoto Yokoya 10.1 Introduction 136 10.2 Pansharpening 137 10.2.1 Survey of Pansharpening Methods Employing Deep Learning 137 10.2.2 Experimental Results 140 10.2.2.1 Experimental Design 140 10.2.2.2 Visual and Quantitative Comparison in Pansharpening 140 10.3 Multiband Image Fusion 143 10.3.1 Supervised Deep Learning-based Approaches 143 10.3.2 Unsupervised Deep Learning-based Approaches 145 10.3.3 Experimental Results 146 10.3.3.1 Comparison Methods and Evaluation Measures 146 10.3.3.2 Dataset and Experimental Setting 146 10.3.3.3 Quantitative Comparison and Visual Results 147 10.4 Conclusion and Outlook 148 11 Deep Learning for Image Search and Retrieval in Large Remote Sensing Archives 150Gencer Sumbul, Jian Kang, and Begüm Demir 11.1 Introduction 150 11.2 Deep Learning for RS CBIR 152 11.3 Scalable RS CBIR Based on Deep Hashing 156 11.4 Discussion and Conclusion 159 Acknowledgement 160 Part II Making a Difference in the Geosciences with Deep Learning 161 12 Deep Learning for Detecting Extreme Weather Patterns 163Mayur Mudigonda, Prabhat Ram, Karthik Kashinath, Evan Racah, Ankur Mahesh, Yunjie Liu, Christopher Beckham, Jim Biard, Thorsten Kurth, Sookyung Kim, Samira Kahou, Tegan Maharaj, Burlen Loring, Christopher Pal, Travis O'Brien, Kenneth E. Kunkel, Michael F. Wehner, and William D. Collins 12.1 Scientific Motivation 163 12.2 Tropical Cyclone and Atmospheric River Classification 166 12.2.1 Methods 166 12.2.2 Network Architecture 167 12.2.3 Results 169 12.3 Detection of Fronts 170 12.3.1 Analytical Approach 170 12.3.2 Dataset 171 12.3.3 Results 172 12.3.4 Limitations 174 12.4 Semi-supervised Classification and Localization of Extreme Events 175 12.4.1 Applications of Semi-supervised Learning in Climate Modeling 175 12.4.1.1 Supervised Architecture 176 12.4.1.2 Semi-supervised Architecture 176 12.4.2 Results 176 12.4.2.1 Frame-wise Reconstruction 176 12.4.2.2 Results and Discussion 178 12.5 Detecting Atmospheric Rivers and Tropical Cyclones Through Segmentation Methods 179 12.5.1 Modeling Approach 179 12.5.1.1 Segmentation Architecture 180 12.5.1.2 Climate Dataset and Labels 181 12.5.2 Architecture Innovations: Weighted Loss and Modified Network 181 12.5.3 Results 183 12.6 Challenges and Implications for the Future 184 12.7 Conclusions 185 13 Spatio-temporal Autoencoders in Weather and Climate Research 186Xavier-Andoni Tibau, Christian Reimers, Christian Requena-Mesa, and Jakob Runge 13.1 Introduction 186 13.2 Autoencoders 187 13.2.1 A Brief History of Autoencoders 188 13.2.2 Archetypes of Autoencoders 189 13.2.3 Variational Autoencoders (VAE) 191 13.2.4 Comparison Between Autoencoders and Classical Methods 192 13.3 Applications 193 13.3.1 Use of the Latent Space 193 13.3.1.1 Reduction of Dimensionality for the Understanding of the System Dynamics and its Interactions 195 13.3.1.2 Dimensionality Reduction for Feature Extraction and Prediction 199 13.3.2 Use of the Decoder 199 13.3.2.1 As a Random Sample Generator 201 13.3.2.2 Anomaly Detection 201 13.3.2.3 Use of a Denoising Autoencoder (DAE) Decoder 202 13.4 Conclusions and Outlook 203 14 Deep Learning to Improve Weather Predictions 204Peter D. Dueben, Peter Bauer, and Samantha Adams 14.1 Numerical Weather Prediction 204 14.2 How Will Machine Learning Enhance Weather Predictions? 207 14.3 Machine Learning Across the Workflow of Weather Prediction 208 14.4 Challenges for the Application of ML in Weather Forecasts 213 14.5 The Way Forward 216 15 Deep Learning and the Weather Forecasting Problem: Precipitation Nowcasting 218Zhihan Gao, Xingjian Shi, Hao Wang, Dit-Yan Yeung, Wang-chun Woo, and Wai-Kin Wong 15.1 Introduction 218 15.2 Formulation 220 15.3 Learning Strategies 221 15.4 Models 223 15.4.1 FNN-based Odels 223 15.4.2 RNN-based Models 225 15.4.3 Encoder-forecaster Structure 226 15.4.4 Convolutional LSTM 226 15.4.5 ConvLSTM with Star-shaped Bridge 227 15.4.6 Predictive RNN 228 15.4.7 Memory in Memory Network 229 15.4.8 Trajectory GRU 231 15.5 Benchmark 233 15.5.1 HKO-7 Dataset 234 15.5.2 Evaluation Methodology 234 15.5.3 Evaluated Algorithms 235 15.5.4 Evaluation Results 236 15.6 Discussion 236 Appendix 238 Acknowledgement 239 16 Deep Learning for High-dimensional Parameter Retrieval 240David Malmgren-Hansen 16.1 Introduction 240 16.2 Deep Learning Parameter Retrieval Literature 242 16.2.1 Land 242 16.2.2...