Difference between revisions of "Creating Maps in Jupyter Notebook using GeoPandas"
(→Data) |
|||
(61 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
==Introduction== |
==Introduction== |
||
− | === |
+ | ===Purpose=== |
− | This tutorial will demonstrate how Jupyter Notebook can be used to manage and display spatial data in conjunction with Anaconda |
+ | This tutorial will demonstrate how Jupyter Notebook can be used to manage and display spatial data in conjunction with Anaconda and GeoPandas. Using Jupyter Notebook in this procedure will enable you to develop your skills in scripting and automatic mapping. This workflow is accessible to users with less computational power, and data will be easier to store and manage. In the second part of this tutorial, we will show you how to display the data as a complete map (utilizing basic cartographic elements). |
− | This tutorial is designed for GIS users with some experience using graphic user interfaces, who are looking to get into using Python to streamline their workflow. The instructions and figures included in this tutorial were developed on the Windows operating system. If you are using a different operating system, your process may be slightly different. |
+ | This tutorial is designed for GIS users with some experience using graphic user interfaces, who are looking to get into using Python to streamline their workflow. The instructions and figures included in this tutorial were developed on the Windows operating system. If you are using a different operating system, your process may be slightly different. |
===About Jupyter=== |
===About Jupyter=== |
||
Jupyter Notebook was created by Project Jupyter, which is a collective which aims to develop open-source software in various programming languages. Jupyter Notebook specifically enables users to easily create and share code, as well as visualise data, among other uses. It is free to download and use (Jupyter, n.d.). While it runs in web browsers, it also runs locally on the user’s machine, which makes it easy to save version controls locally. |
Jupyter Notebook was created by Project Jupyter, which is a collective which aims to develop open-source software in various programming languages. Jupyter Notebook specifically enables users to easily create and share code, as well as visualise data, among other uses. It is free to download and use (Jupyter, n.d.). While it runs in web browsers, it also runs locally on the user’s machine, which makes it easy to save version controls locally. |
||
Another advantage of Jupyter is that you can type code into kernels, and run those kernels individually. This will be demonstrated in the tutorial, but as a quick explanation -- the benefits to this include being able to test code easily and quickly visualize a certain data table, or in our case, create a map quickly without having to run all of the code at once. |
Another advantage of Jupyter is that you can type code into kernels, and run those kernels individually. This will be demonstrated in the tutorial, but as a quick explanation -- the benefits to this include being able to test code easily and quickly visualize a certain data table, or in our case, create a map quickly without having to run all of the code at once. |
||
− | The main strength of Jupyter that we will be highlighting in this tutorial is its utility in visualising data that is updated frequently. The most pertinent example we have of that today are the daily updates in cases of COVID-19, which have been very effectively communicated through the [https://coronavirus.jhu.edu/map.html use of maps]. |
||
− | ===About GeoPandas |
+ | ===About GeoPandas=== |
[https://geopandas.org/ GeoPandas] is an open source library in Jupyter that builds off of [https://pandas.pydata.org/ pandas] in Python. It is designed for users to more easily perform geospatial operations, by taking advantage of data frames in pandas, and creating spatial data frames. GeoPandas uses [https://fiona.readthedocs.io/en/latest/ fiona] for accessing files, [https://pypi.org/project/Shapely/ Shapely] objects for geometric manipulation and [https://matplotlib.org/ Matplotlib] for plotting (geoPandas, n.d.). |
[https://geopandas.org/ GeoPandas] is an open source library in Jupyter that builds off of [https://pandas.pydata.org/ pandas] in Python. It is designed for users to more easily perform geospatial operations, by taking advantage of data frames in pandas, and creating spatial data frames. GeoPandas uses [https://fiona.readthedocs.io/en/latest/ fiona] for accessing files, [https://pypi.org/project/Shapely/ Shapely] objects for geometric manipulation and [https://matplotlib.org/ Matplotlib] for plotting (geoPandas, n.d.). |
||
− | [https://pypi.org/project/Cartopy/ CartoPy] is an open source package in Python that is designed for geospatial data processing. It takes advantage of shapely and [https://numpy.org/ NumPy] libraries, and also uses matplotlib for plotting. It easily transforms points, lines and polygons based on geospatial projections, which helps virtualize and visualize data. |
||
− | |||
− | ===About QGIS=== |
||
− | QGIS is a free and open-source comprehensive desktop GIS software with a wide variety of features that facilitate the display, analysis, and publishing of spatial data (QGIS, n.d.). In the context of this tutorial, QGIS enables us to expand on the work done in Jupyter by harnessing QGIS’ cartographic tools to produce a finished map with basic cartographic elements that can be saved and reused on updated iterations of the data. |
||
===Note on Software Versions=== |
===Note on Software Versions=== |
||
− | This tutorial uses the latest versions of software available at the time of writing ( |
+ | This tutorial uses the latest versions of software available at the time of writing (October 2024). Python 3.12, and geoPandas 0.14.1. If you find updated versions of software when you try this tutorial, please note that there may be some differences in what you see in our screenshots and instructions, and what you see on your software. |
==Part 1: Getting Started== |
==Part 1: Getting Started== |
||
===Software=== |
===Software=== |
||
− | This tutorial will be for Windows machines. The following steps will assume that the user is on a Windows platform, and therefore if you are using any other machine, the steps may be slightly different. |
+ | This tutorial will be for Windows machines. The following steps will assume that the user is on a Windows platform, and therefore if you are using any other machine, the steps may be slightly different. This tutorial also assumes that the user already has QGIS installed. If you do not have QGIS already, please go to [https://www.qgis.org/en/site/ this website] to do so. |
====''Install Anaconda''==== |
====''Install Anaconda''==== |
||
+ | Install Anaconda [https://docs.anaconda.com/anaconda/install/ here].<br> |
||
− | Install Anaconda [https://docs.anaconda.com/anaconda/install/ here]. You may also choose to simply install miniconda which will run everything necessary for the purpose of this tutorial, and most Jupyter notebook commands. You can install miniconda [https://docs.anaconda.com/anaconda/install/silent-mode/ here] in silent mode (recommended). Silent mode will automatically accept default settings and allow for quicker installation. |
||
+ | Follow the images to install Anaconda then open the Anaconda Navigator and Install/Launch JupyterLab<br> |
||
+ | === Step 1. === |
||
+ | [[File:Installation Step 1.png|1000px]]<br> |
||
+ | === Step 2. === |
||
+ | [[File:Installation Step 2.png|1000px]]<br> |
||
+ | === Step 3. === |
||
+ | [[File:Installation Step 3.png|1000px]]<br> |
||
+ | === Step 4. === |
||
+ | After Opening Anaconda Navigator <br> |
||
+ | [[File:Installation Step 4.png|1000px]]<br> |
||
+ | ===Data=== |
||
− | ====''Install Jupyter Notebook''==== |
||
+ | All the data and code can be downloaded from this [https://drive.google.com/drive/folders/1sgWFVFYaXLO59QSRAuFco0fXgBpllPTs?usp=sharing Google Drive Link] <br> |
||
− | Next, you will need to install Jupyter Notebook.This can be done easiest through the Command Line in Windows. To open this up, you can search for “Anaconda Prompt” in the Windows Start Menu, and right-click “Run as Administrator” on the Anaconda Command Prompt application. |
||
+ | The download contains: <br> |
||
+ | * The Jupyter Source File<br> |
||
+ | * The Final Map in both PNG and PDF formats |
||
+ | * The Census Area Population Data<ref>https://opendata.vancouver.ca/explore/dataset/census-local-area-profiles-2016/information/]</ref> |
||
+ | * The local-area-boundary Shapefile<ref>https://opendata.vancouver.ca/explore/dataset/local-area-boundary/map/?disjunctive.name&location=13,49.2474,-123.12402</ref> |
||
+ | ==Part 2: Opening the Code== |
||
− | [[File:Anacondafig1.jpg|300px]] |
||
+ | Once JupyterLab is open, navigate to the downloaded folder using the file browser tab on the left side. Once inside double click on ''MappingInJupterLab.ipynb'' and follow along <br> |
||
− | |||
+ | [[File:JupyterLab File Path.png]] |
||
− | <small>'''Figure 1.''' How to open Anaconda Prompt.</small> |
||
− | |||
− | Install Jupyter Notebook in your users file directory by “Changing Directory” using cd to where you want to install it and using the following command: |
||
− | |||
− | ''conda install -c conda-forge jupyterlab'' |
||
− | |||
− | ====''Install GeoPandas''==== |
||
− | Install the GeoPandas library using the same Anaconda Prompt application as used before. You should “Change Directory” using the cd prompt to where you would like geopandas to be installed. For the purpose of this tutorial, I will be installing it into a working folder, however I recommend installing both Jupyter Notebook and GeoPandas somewhere permanent in your file directory that will be easy to navigate to in the future, for example your users folder. Once in your desired file directory, install the latest version of GeoPandas using: |
||
− | |||
− | ''conda install geopandas'' |
||
− | |||
− | [[File:Geopandasfig2.jpg|500px]] |
||
− | |||
− | <small>'''Figure 2. ''' How to install GeoPandas.</small> |
||
− | |||
− | Next, we will be creating a new environment for geopandas. This is optional, but also recommended as good practise, as you may have dependency conflicts from previous installs of other software on your machine. This way, we can have a fresh start. To do this, enter the following command: |
||
− | |||
− | ''conda create --name [name of environment]'' |
||
− | |||
− | (Note: -n and --name is the same syntax) |
||
− | |||
− | Once created, you can activate this environment using: |
||
− | |||
− | ''conda activate [name of environment]'' |
||
− | |||
− | [[File:GeopandasFig3.jpg|500px]] |
||
− | |||
− | <small>'''Figure 3. ''' Setting up the GeoPandas environment.</small> |
||
− | |||
− | Next, configure the environment to work with packages and install GeoPandas within them. Use the following commands, accepting the defaults with '''y'''. |
||
− | |||
− | ''conda config --env --add channels conda-forge'' |
||
− | |||
− | ''conda config --env --set channel_priority strict'' |
||
− | |||
− | ''conda install python=3 geopandas'' |
||
− | |||
− | ====''Install Matplotlib and CartoPy''==== |
||
− | These packages will allow us to plot. Install them by running: |
||
− | |||
− | ''conda install -c conda-forge matplotlib'' |
||
− | |||
− | ''conda install -c conda-forge cartopy'' |
||
− | |||
− | ''conda install -c conda-forge descartes'' |
||
− | |||
− | ===Data=== |
||
− | Download the following data and save the .csv files into your working directory: |
||
+ | ==Part 3: Tools and Data Organization== |
||
− | ''John Hopkins COVID-19 Dataset:'' |
||
+ | ===Tools=== |
||
+ | Here is the code cell that contains all the import statements that will be used in the tutorial<br> |
||
+ | <br> |
||
+ | [[File:Import Statements.png]]<br> |
||
+ | <br> |
||
+ | * '''import pandas as pd:''' |
||
+ | :This imports the pandas library, a powerful tool for data manipulation and analysis, particularly with tabular data (dataframes). The alias pd is commonly used for brevity. |
||
+ | * '''import numpy as np:''' |
||
+ | :This imports NumPy, a library used for numerical computations, especially with arrays and matrices. In our case we use it for the orientation of the North Arrow. |
||
+ | * '''import matplotlib.pyplot as plt:''' |
||
+ | :This imports pyplot from the matplotlib library, which is widely used for creating static, animated, and interactive visualizations in Python. The alias plt is used to simplify plotting commands. |
||
+ | * '''import geopandas as gpd:''' |
||
+ | :This imports GeoPandas, an extension of pandas that adds support for geospatial data. It simplifies working with geographical data, such as handling shapefiles and performing spatial operations. |
||
+ | * '''import os:''' |
||
+ | :This imports Python’s built-in os module, which provides functionality for interacting with the operating system, such as reading or writing files, navigating directories, and managing paths. |
||
+ | ===Importing The CSV=== |
||
− | [https://covid19.who.int/table Data Table] |
||
+ | This is the code cell that imports the CSV file into a dataframe.<br> |
||
+ | [[File:Importing CSV.png]]<br> |
||
+ | * Note that there is ''encoding='ISO-8859-1''' which is required because of the CSV's file formatting, in most cases an encoding will not be necessary<br> |
||
+ | * It is always a good idea to print out the data you have just imported to make sure that it imported correct<br> |
||
+ | :* the ''print(df.head())'' will print the first 5 rows of the CSV<br> |
||
+ | ===Converting CSV to Desired Formatting=== |
||
− | ==Part 2: Jupyter Notebook== |
||
+ | One of the biggest strengths of using Python in your workflow is the ability to automate tasks. The CSV file contains lots of information we do not need and the columns and rows are inverted to what would be intuitive. The following cell shows all the modifications and moves that were made in order to get the Are names in one column with their matching populations in the column to the right. If you want more details on what each of the lines of code is doing read the commends in the code. Once again we are printing the result to make sure they are in the desired formatting. <br> |
||
+ | [[File:Converting CSV to Desired Formatting.png]]<br> |
||
+ | ===Importing The Shapefile=== |
||
+ | This cell takes the shapefile and imports it into a geodataframe. Note that we are printing the columns with ''print(gdf.columns)'' because we need to know the names of the columns we will be merging data to.<br> |
||
+ | [[File:Importing Shapefile.png]]<br> |
||
− | ==Part |
+ | ==Part 4: Merging== |
+ | The next step is the merge the data onto the geodataframe using a common variable. In this case we will must the name columns of both data. The ''how='left''' clarifies that we are adding the population data to the left dataset (being the shapefile)<br> |
||
+ | [[File:Merging Cell.png]] |
||
+ | ==Part 5: Displaying== |
||
− | ==Conclusion== |
||
+ | Here is the cell that displays the finalized map. There are lots of different customization options that are explained in the code comments.<br> |
||
+ | [[File:Displaying2.png]]<br> |
||
+ | We are using matplotlib which is primarily used for graphs so there are little changes that we have to make to a good map. Additionally, there is no good implementations of a north arrow so we are forced to make our own. For the vast majority of maps the north arrow will be straight up but in the odd case where it is not you will have to manually change the angle of the arrow. Right before the map is displayed there are two commented lines of code which save the final product to either a png or a pdf. This is very useful if you want to share what you have made to others. |
||
− | == |
+ | ==Final Result== |
+ | [[File:Population by Area in Vancouver.png]]<br> |
Latest revision as of 15:16, 4 October 2024
Contents
Introduction
Purpose
This tutorial will demonstrate how Jupyter Notebook can be used to manage and display spatial data in conjunction with Anaconda and GeoPandas. Using Jupyter Notebook in this procedure will enable you to develop your skills in scripting and automatic mapping. This workflow is accessible to users with less computational power, and data will be easier to store and manage. In the second part of this tutorial, we will show you how to display the data as a complete map (utilizing basic cartographic elements). This tutorial is designed for GIS users with some experience using graphic user interfaces, who are looking to get into using Python to streamline their workflow. The instructions and figures included in this tutorial were developed on the Windows operating system. If you are using a different operating system, your process may be slightly different.
About Jupyter
Jupyter Notebook was created by Project Jupyter, which is a collective which aims to develop open-source software in various programming languages. Jupyter Notebook specifically enables users to easily create and share code, as well as visualise data, among other uses. It is free to download and use (Jupyter, n.d.). While it runs in web browsers, it also runs locally on the user’s machine, which makes it easy to save version controls locally. Another advantage of Jupyter is that you can type code into kernels, and run those kernels individually. This will be demonstrated in the tutorial, but as a quick explanation -- the benefits to this include being able to test code easily and quickly visualize a certain data table, or in our case, create a map quickly without having to run all of the code at once.
About GeoPandas
GeoPandas is an open source library in Jupyter that builds off of pandas in Python. It is designed for users to more easily perform geospatial operations, by taking advantage of data frames in pandas, and creating spatial data frames. GeoPandas uses fiona for accessing files, Shapely objects for geometric manipulation and Matplotlib for plotting (geoPandas, n.d.).
Note on Software Versions
This tutorial uses the latest versions of software available at the time of writing (October 2024). Python 3.12, and geoPandas 0.14.1. If you find updated versions of software when you try this tutorial, please note that there may be some differences in what you see in our screenshots and instructions, and what you see on your software.
Part 1: Getting Started
Software
This tutorial will be for Windows machines. The following steps will assume that the user is on a Windows platform, and therefore if you are using any other machine, the steps may be slightly different. This tutorial also assumes that the user already has QGIS installed. If you do not have QGIS already, please go to this website to do so.
Install Anaconda
Install Anaconda here.
Follow the images to install Anaconda then open the Anaconda Navigator and Install/Launch JupyterLab
Step 1.
Step 2.
Step 3.
Step 4.
After Opening Anaconda Navigator
Data
All the data and code can be downloaded from this Google Drive Link
The download contains:
- The Jupyter Source File
- The Final Map in both PNG and PDF formats
- The Census Area Population Data[1]
- The local-area-boundary Shapefile[2]
Part 2: Opening the Code
Once JupyterLab is open, navigate to the downloaded folder using the file browser tab on the left side. Once inside double click on MappingInJupterLab.ipynb and follow along
Part 3: Tools and Data Organization
Tools
Here is the code cell that contains all the import statements that will be used in the tutorial
- import pandas as pd:
- This imports the pandas library, a powerful tool for data manipulation and analysis, particularly with tabular data (dataframes). The alias pd is commonly used for brevity.
- import numpy as np:
- This imports NumPy, a library used for numerical computations, especially with arrays and matrices. In our case we use it for the orientation of the North Arrow.
- import matplotlib.pyplot as plt:
- This imports pyplot from the matplotlib library, which is widely used for creating static, animated, and interactive visualizations in Python. The alias plt is used to simplify plotting commands.
- import geopandas as gpd:
- This imports GeoPandas, an extension of pandas that adds support for geospatial data. It simplifies working with geographical data, such as handling shapefiles and performing spatial operations.
- import os:
- This imports Python’s built-in os module, which provides functionality for interacting with the operating system, such as reading or writing files, navigating directories, and managing paths.
Importing The CSV
This is the code cell that imports the CSV file into a dataframe.
- Note that there is encoding='ISO-8859-1' which is required because of the CSV's file formatting, in most cases an encoding will not be necessary
- It is always a good idea to print out the data you have just imported to make sure that it imported correct
- the print(df.head()) will print the first 5 rows of the CSV
- the print(df.head()) will print the first 5 rows of the CSV
Converting CSV to Desired Formatting
One of the biggest strengths of using Python in your workflow is the ability to automate tasks. The CSV file contains lots of information we do not need and the columns and rows are inverted to what would be intuitive. The following cell shows all the modifications and moves that were made in order to get the Are names in one column with their matching populations in the column to the right. If you want more details on what each of the lines of code is doing read the commends in the code. Once again we are printing the result to make sure they are in the desired formatting.
Importing The Shapefile
This cell takes the shapefile and imports it into a geodataframe. Note that we are printing the columns with print(gdf.columns) because we need to know the names of the columns we will be merging data to.
Part 4: Merging
The next step is the merge the data onto the geodataframe using a common variable. In this case we will must the name columns of both data. The how='left' clarifies that we are adding the population data to the left dataset (being the shapefile)
Part 5: Displaying
Here is the cell that displays the finalized map. There are lots of different customization options that are explained in the code comments.
We are using matplotlib which is primarily used for graphs so there are little changes that we have to make to a good map. Additionally, there is no good implementations of a north arrow so we are forced to make our own. For the vast majority of maps the north arrow will be straight up but in the odd case where it is not you will have to manually change the angle of the arrow. Right before the map is displayed there are two commented lines of code which save the final product to either a png or a pdf. This is very useful if you want to share what you have made to others.