Spatial Pattern Analysis with CartoDB

From CUOSGwiki
Jump to navigationJump to search

Introduction to CartoDB

For many general GIS users CartoDB may seem complex and very daunting. But the great thing with CartoDB is that being a master at SQL, CSS and databases is not always required for using CartoDB. For more basic use cases it can be used with limited coding experience. It is extremely powerful and versatile with a wide array of use cases, namely data visualization, spatial analysis, and other geospatial applications.

In this tutorial it will provide a basic run down of how to set up CartoDB and provide some information on performing spatial analysis in CartoDB.

Install Guide for Ubuntu 20.04 LTS x64

In this tutorial it will guide you through the basic process of installing CartoDB on Ubuntu 20.04. Most will likely install CartoDB inside a VM on a local PC but other commercial options exist. The creators of CartoDB have a basic, 12 month free trial and paid options. Alternative hosting options are also available from Digital Ocean.

CartoDB has several dependencies which it uses to run:

  • PostgreSQL
  • PostGIS
  • Redis
  • CARTO PostgreSQL extensions
  • CARTO Builder
  • Maps API
  • SQL API

Install GIT

This tutorial requires the use of git for the installation of some packages.

sudo apt-get install git

Install PostgreSQL

Add the custom PostgreSQL repository for CartoDB. The reason for the custom package can be found here.

sudo add-apt-repository ppa:cartodb/postgresql-10 && sudo apt-get update

Then install PostgreSQL

sudo apt-get install postgresql-10 \
postgresql-plpython-10 \
postgresql-server-dev-10

PostgreSQL Configuration

To make this installation easier, and because we are installing within a local environment we can make authentication less secure.

Simply run:

sudo nano etc/postgresql/10/main/pg_hba.conf

And make sure that the lines appear as below:

local all postgres trust
local all all trust
host all all 127.0.0.1/32 trust

Once done restart PostgreSQL:

sudo systemctl restart postgresql

Then add the required CARTO users to PostgreSQL:

sudo createuser publicuser --no-createrole --no-createdb --no-superuser -U postgres
sudo createuser tileuser --no-createrole --no-createdb --no-superuser -U postgres

Install PostgreSQL Helper Extensions

Finally install some PostgreSQL extensions that expand upon PostgreSQL to work better with the other dependencies.

git clone https://github.com/CartoDB/cartodb-postgresql.git
cd cartodb-postgresql
git checkout <LATEST cartodb-postgresql tag>
sudo make all install

Install GDAL and PostGIS

sudo add-apt-repository ppa:cartodb/gis && sudo apt-get update
sudo apt-get install gdal-bin libgdal-dev postgis

Configure PostGIS Database:
sudo createdb -T template0 -O postgres -U postgres -E UTF8 template_postgis
psql -U postgres template_postgis -c 'CREATE EXTENSION postgis;CREATE EXTENSION postgis_topology;'
sudo ldconfig

Install Redis

sudo add-apt-repository ppa:cartodb/redis-next && sudo apt-get update
sudo apt-get install redis

Install Node.js

curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
sudo apt-get install -y nodejs

Some more dependencies:
sudo apt-get install libpixman-1-0 libpixman-1-dev libcairo2-dev libjpeg-dev libgif-dev libpango1.0-dev

I promise we're getting close to the end.

Install SQL API

git clone git://github.com/CartoDB/CartoDB-SQL-API.git
cd CartoDB-SQL-API
npm install

Configure our local development environment and then Start Node.js:
cp config/environments/development.js.example config/environments/development.js
node app.js development

Install MAPS API

git clone git://github.com/CartoDB/Windshaft-cartodb.git
cd Windshaft-cartodb
npm install

Then configure the API and start it:
cp config/environments/development.js.example config/environments/development.js
mkdir logs
node app.js development

Install Ruby

  • No not the mineral type

sudo apt-add-repository ppa:brightbox/ruby-ng && sudo apt-get update
sudo apt-get install ruby2.4 ruby2.4-dev ruby-bundler
sudo gem install compass

And Finally Install Builder and CartoDB

Download and install CartoDB with additonal dependencies:
git clone --recursive https://github.com/CartoDB/cartodb.git
cd cartodb
sudo apt-get install python-pip imagemagick unp zip libicu-dev
RAILS_ENV=development bundle install
sudo pip install --no-use-wheel -r python_requirements.txt

Finish up:
npm install
npm run carto-node && npm run build:static

Configure CartoDB:
cp config/app_config.yml.sample config/app_config.yml
cp config/database.yml.sample config/database.yml

Start remaining services and initialize database:
sudo systemctl start redis-server
RAILS_ENV=development bundle exec rake db:create
RAILS_ENV=development bundle exec rake db:migrate
RAILS_ENV=development bundle exec rails server

And finally close that console and then in a new one run:
RAILS_ENV=development bundle exec ./script/resque

First Time CartoDB Run

Create user account and development environment:
cd cartodb
export SUBDOMAIN=development

# Add entries to /etc/hosts in development
echo "127.0.0.1 ${SUBDOMAIN}.localhost.lan" | sudo tee -a /etc/hosts

# Create the development user
sh script/create_dev_user

Run the remaining processes:
bundle exec script/resque
bundle exec thin start --threaded -p 3000 --threadpool-size 5
cd cartodb-sql-api && node app.js
cd windshaft-cartodb && node app.js

That's it!
Go to http://<mysubdomain>.localhost.lan:3000 And enter your login created using the password entered above.

User Interface

Now that CartoDB is intstalled there are several ways to begin using it.
The three most popular ones that many people use are directly throught the web builder interface online, through Jupiter Notebook or with the PythonSDK.

How to upload data?

Importing data is seamless and easy. The 'new table' icon with the plus sign will open a dialog box with options to:

  • paste a url or select a file (e.x. shapefiles),
  • Online cloud storage such as dropbox or google drive data,
  • or create a new table from scratch or other database services

How to explore data?

Each dataset is known as a table that corresponds to a spreadsheet that can be visualized on a map. Examination or modification of data are performed by SQL statements or using the GUI.

How to map data? Styling, Visualization Wizard, Base maps, Labels

Cartodb provides full control over styling of map using SQL and CSS. Visualization wizards provide options to visualize data with simple, choropleth, category, bubble, intensity, density and a animated data categorization called torque. Torque, simple and category are used in this project. Column labels are toggled on for each row to provide the user information about name and location for incubators and bixi docks with the addition of how many bikes are available at each location. There are support from base maps from Google, CartoDB, Mapbox, WMS, XYZ and more. CartoDB Dark is used in this project.

Many great examples of projects created with CartoDB and further video tutorials or help centers exist online.

How to share the map?

Maps are shared from the options menu in the top right, by link, embed or as an api.

Dataset

Bicycle Stations (Bixi) and Business Incubators from the City of Toronto open data initiative are the two data sets utilized in this project. As of this writing, business incubators is no longer offered. Bixi dataset had been preprocessed to include the latitude and longitude derived from address.

Data Analysis

Cartodb has full support for both vector and raster data. The example below takes advantage of a short SQL script.

Example: SQL statement is derived from Andrew Hill: https://gist.github.com/andrewxhill/5979532. 

 SELECT
 ST_MakeLine( --This function can take two or more points and make a line
  the_geom_webmercator, --We select the_geom_webmercator, since CartoDB will need it to draw your maps
  ( --This is a nested query that will run for every row in our outer query
    SELECT the_geom_webmercator FROM plout10 -- Here we select the geometry from our second dataset
    ORDER BY the_geom <-> c.the_geom -- We then order it by its distance to the geometry in the first dataset (c.the_geom)
    LIMIT 1  -- And limit it to just 1, i.e. we find just the closest
    )
    ) the_geom_webmercator -- Here we alias the result to a column we call, 'the_geom_webmercator', so that CartoDB will draw it
  FROM citibike_stations c -- Here we alias our table to 'c' so we can type it nicely above :)


The following SQL statement is used to visualize the closest Biki locations to an incubator. These statement utilizes PostGIS and PostgresSQL functions. 

 SELECT 
 ST_MakeLine(the_geom_webmercator, 
   ( 
     SELECT the_geom_webmercator FROM business_incubators_toronto 
     ORDER BY the_geom <-> c.the_geom
     LIMIT 1  
   )
  ) the_geom_webmercator
  FROM station c

References

VirtualBox https://www.virtualbox.org/
Old Ubuntu Releases http://old-releases.ubuntu.com/releases/
Digital Ocean https://www.digitalocean.com/
CartoDB Repository https://github.com/CartoDB/cartodb
PostGIS Reference http://postgis.net/docs/reference.html
PostgreSQL Tutorial http://www.postgresql.org/docs/9.1/static/tutorial.html
Lord Linus's RVM tutorial https://github.com/lordlinus/cartodb
Michael Schmid's suggestions https://groups.google.com/forum/#!topic/cartodb/o5_cVk-owe0
Andrew Hill's SQL query https://gist.github.com/andrewxhill/5979532
Open Data Toronto http://www1.toronto.ca/wps/portal/contentonly?vgnextoid=1a66e03bb8d1e310VgnVCM10000071d60f89RCRD
Bixi http://www1.toronto.ca/wps/portal/contentonly?vgnextoid=ad3cb6b6ae92b310VgnVCM10000071d60f89RCRD&vgnextchannel=1a66e03bb8d1e310VgnVCM10000071d60f89RCRD

Retrieved from "https://dges.carleton.ca/CUOSGwiki/index.php?title=Spatial_Pattern_Analysis_with_CartoDB&oldid=15676"