Time Series Analysis using Google Earth Engine and Google Colab
Contents
Introduction/Earth Engine Background
Google Earth Engine is a powerful and useful platform with many built-in tools to run geospatial analyses using Google's servers and computational infrastructure. Earth Engine hosts a number of datasets of various categories and applications including selections of Landsat collections from 1979 to present, Sentinel collections, MODIS collections, and a host of various data including DEMs, crop inventories, several radiance sensors, and many more vector and raster datasets.
This data can be accessed, queried, and processed within a short timeframe and allows a user to select and process data into workable products within minutes. Google Earth Engine relies on user scripting to query datasets and process data into charts, graphs, maps, animations, applets, and any available products that Google's API offers. This tutorial will assume that you have an understanding of Python scripting and GIS experience, however all code will be provided.
Earth Engine offers a 'Code Editor' and 'Explorer' to find, access, and manipulate data. The Code Editor is used in strictly JavaScript API/language, an example of which can be seen at Supervised Classifications using Google Earth Engine by Loucas Diamant-Boustead. Rather than using the Code Editor exclusively, Google also offers a Python API within the Google Colab environment. Colab allows for instant Python writing and execution within a web browser, with little setup required. There are plenty of Python libraries that can be used to assist in geospatial analysis, with the ability to create, query, and mutate dataframes (like Excel), perform spatial joins, perform statistical analysis, and to map and plot trends using Earth Engine datasets.
This tutorial will focus on using Google Colab as a platform to perform Earth Engine analyses. Python functions and processes will be displayed as code and explained at a basic level, with a focus placed on the meaning/reasoning behind the code, rather than a full explanation of how Python scripting works. Basic levels of knowledge and understanding of GIS, statistics, and Python is highly recommended for this tutorial.
The subject of this walkthrough will focus on Google Earth Engine's Sentinel 5-P TROPOMI dataset, which is offered at a level 3 (pre-processed) product. Because it is pre-processed, its data products can be used directly to perform analysis. No pre-processing or quality assurance work is necessary (as is necessary with other Level 1-2 remote sensing datasets). Google Earth Engine's Colab API will be utilized to analyze changes in atmospheric nitrogen dioxide (NO2) during the 2020 COVID pandemic (specifically, their related lockdowns) using TROPOMI products. An example of both statistical/quantitative analysis and visual/qualitative analysis will be provided and walked through.
Initial Setup
Signing up for an Earth Engine Account
A key part in using Google Earth Engine for any purpose, either in Colab or the Code Editor, is setting up an account and applying for Earth Engine access.
Although Google Earth Engine is an 'open' platform, it does require that you fill out a form to setup an Earth Engine account. As well, you'll need to create a Google-platform account in order to access Earth Engine and Google Colab notebooks/Google Drive, so the preliminary step would be to sign-in or sign up for a Google Account on the Google platform.
With a Google account, you can apply for an Earth Engine account here. The setup process will be detailed below.
Steps to the setup process:
- If you haven't already, create a Google platform account (with any type of email) at Google.ca (top right corner of page).
- Go to this website and sign-up for an Earth Engine account
- Add your email (this will be done automatically if signed in to Google).
- Add your first and last name (don't lie here!)
- Add your institution (likely Carleton University) and type of institution (likely Education/Academic).
- Select your region (likely Canada)
- Provide a few sentence description of why you want to use Earth Engine
- Accept the Terms of Service
- Do not apply for a commercial-use license
- Prove you are not a robot
- You will likely receive an email of your successful registration within 24 hours.
And that's all it takes to sign-up for an Earth Engine account! It usually is a very simple process, and if you are denied you can re-apply. The most common reason for denial would be incorrect information, or the accidental application for a commercial license.
Setting up your Google Colab Environment
The first step in setting up a Colaboratory (Colab) environment is to go to this website.
This page will also provide you with some background on what Colab is, what it is used for, its advantages, some basic information and demonstrations, and the various applications of the environment and application. Reading this page is highly suggested to learn more about Colab and how it uses Python in an interactive notebook environment, as well as providing useful links for how-tos and demonstrations showing how and what Colab can do.
Once you have read through the 'splash' page for Google Colab, you can create your own Colab notebook by clicking 'File/New notebook' to create your own, blank Python notebook.
Within the Colab notebook environment (and within your new notebook), you can add two types of cells to your notebook:
- Markdown Cells are used to add formatted text, headings, sub-headings, notes and anything that isn't Python script to your notebook. In this tutorial, we won't be using many markdown cells (some may be used for images/illustrations), and they aren't relevant to the process of strict geospatial analysis within this environment. They are quite useful, however, for interactivity, whether using them as the basis for a tutorial (rather than a Wiki page or website), a laboratory for academic purposes (they can be used as text for a lab-writer, or textual answer blocks for a lab student), or other uses, they can be used to illustrate actions being performed in Python in plain English.
- Code Cells allow you to write and execute Python code/scripts within your web browser, which provides a powerful tool for using Earth Engine data in a new environment. Python has plenty of built-in geostatistical libraries, including Pandas, Plotnine (ggplot from R), Altair, matplotlib, scipy, and many more. Others have also made Earth-Engine specific python libraries (specifically geemap, which builds on Colab's Earth Engine mapping methods) that greatly increase the usefulness and user-friendliness of Python/Colab as an Earth Engine tool.
For each step of this tutorial, you can create your own 'Code' cell to coincide with the code that will be written in
a code block, like this.
Installing Python Libraries and Authenticating your Earth Engine Access
Once you have your own, blank Colab notebook, you can begin this exercise by importing relevant Python libraries, and authenticating your Earth Engine session. You'll have to authenticate every time you open the notebook (but only have to authenticate once when starting up). The most important library, called geemap, will have to be downloaded first as seen below. In order to import or install libraries, create a code cell by clicking '+ Code', and type or copy the code blocks below into them. Once fully filled into the code block, 'run' the code by clicking on the large 'Play' button on the left side of the code block.
# This installs the geemap library, a useful mapping and visualization python library for Earth Engine
!pip install geemap
Useful Tip: You can delete outputs from code cells by clicking the 'trash can' in the top corner of your code cell.
The above block of code will install all the dependencies and core of the geemap library, which is used to map Google Earth Engine in the Python environment. In order to used newly installed packages, you'll have to click 'Restart Runtime'.
With that installed, the rest of the Python libraries necessary are built-in, and only need to be imported. We will import the libraries below. Create a new code cell for each of the code blocks, to keep things separated and tidy.
# Import the necessary libraries
import ee
import numpy as np
import geemap.eefolium as geemap
import pprint
import pandas as pd
import altair as alt
import folium
import plotnine
from scipy import stats
# Set up a 'pretty printer' to print 'nicely' within Python
pp = pprint.PrettyPrinter(depth=3)
Some of these libraries (ee, geemap eefolium, folium) are used for interactive mapping (in a similar way as Google Earth/Maps, using its applet). Others are used for statistics (numpy, pandas, scipy/stats), and for our plots (pprint, altair, plotnine).
The final setup step for our Colab environment is Earth Engine's Authentication system. This is necessary for Earth Engine processes to be available within Colab, and for Earth Engine and Colab to have access to your Google Drive (for read/write purposes). Create another code block and add the following:
# Authenticate and initialize this instance of GEE in Google Colab
## Follow the prompts and fill in authentication code
ee.Authenticate()
ee.Initialize()
Authentication and initialization will provide you with an output that has a link you must follow to sign-in and get an authentication code (see images).
In order to authenticate and initialize this Google Earth Engine instance in Colab, follow the link given in the output section below your code block. Once the prompts are followed at the webpage (sign-in, accepting terms), copy your unique code into the vacant 'Enter verification code' in the first image and press Enter. This will successfully save an authorization token, and then we're ready for analysis!
