Exploring Clustering In QGIS

From CUOSGwiki
Jump to navigationJump to search

Introduction

The purpose of this project is to explore the capabilities of open source software such as QGIS, which is one of many open source softwares used in GEOM 4008. This tutorial will explore vector analysis tools used to create clusters and distinct boundaries. An in-depth approach will breakdown each step for new QGIS users, please feel free to skip any steps you are already familiar with.

Cluster analysis is a practical GIS technique which can group vector points into corresponding clusters based on distance, density, or set number of points, and is fully configurable by the user. This can be helpful if you have a large study extent and you want to geographically divide a variables into multiple areas, such as detecting regions of high painted turtle sightings or identifying crime hotspots. ArcGIS Pro has a larger collection of clustering options, however QGIS still has clustering capabilities which are DBSCAN and K-Means.

Here we will explore a local problem: A City of Ottawa employee on the Traffic Services Division team wants to investigate vehicle collision trends by locating accident clusters to recommend updated patrol divisions and officer coverage based on municipality demand.

Resources

Required Software

To run this analysis, it is best to have the latest version of QGIS.

  • If you have an old version, it is recommended to delete this before proceeding to further steps as having two versions of QGIS is not necessary and takes space on the hard drive.
  • Click on this link and you will see many options on operating systems you can download QGIS on.
  • Select either the fastest (4.2.2) or the more stable (3.44.14) download depending on your preference and it should start downloading.
  • Once it's done downloading, add it to your hard drive (if not already done automatically) and you are ready!

Required Datasets

The data that we will be using will be in two forms: Referencing data so we can visualize where we are and vector data to use for clustering.

Referencing

  • The Roads dataset will be used to visualize the roads, this is important to add as those incidents and vector data points are on roads so it can be a helpful tool to illustrate the incidents in a smaller scale.
  • The Wards dataset is a wards dataset which will be used to see where those incidents are with an Ottawa city border background. All data points will be located within that dataset.

Clustering

  • The Traffic Collisions by Location in 2019 dataset is the data set with all the traffic collisions and their frequencies. Their location information will help us separate them into clusters. Once you have added this dataset, I recommend a change as name as it will automatically save as a complicated alphanumeric sequence.

Environment Setup

Add vector data

  • All datasets can be downloaded by clicking on the cloud icon and selecting Shapefile. Organize these files into a project folder and then extract them.
    Download Data.png


  • Go on QGIS and go to Layer menu, Add Layer and select Add Vector Layer
    QGIS AddLayer.png


  • Under Source, click on the 3 dots, find and select each of your project's downloaded datasets (shp.), and Add.
    Upload Files.png


Projection

Projection is an important step for DBSCAN clustering, you may encounter problems if you do not do it this way. NAD83, specifically UTM zone 18, is an appropriate projection for this project as we are looking at a municipal problem, Ottawa traffic collisions. Projection is important as DBSCAN clustering relies on map distances.

Tip: All vector data should be projected that way.

To first orient yourself, we'll add Google Satellite basemap.

  • Select Plugins, then Manage and Install Plugins..., search "Basemaps", select Basemaps and Install.
    BasemapPlugin.png


  • Reselect Plugins, Basemap Management, Load Basemaps, then choose whichever you feel is best at representing your data. Google Satellite is a common and versatile option. Select your choosing and then Load.
    GoogleImagery.png


Now it's time to set the projection, to do so we will save those features with a new CRS:

  • Right click the Traffic Collisions 2019 layer, select Export then Save Feature As.
    ExportingLayers.png


  • Choose a location in your project folder and name it. Ex: Traffic_Collisions2019_Reprojected.
  • On CRS, click the world icon, search for "EPSG:2959" or "NAD83(CRSR) / UTM zone 18N", select it and press OK.
    Reprojecting Layer.png

Symbology

Symbology is a great visual method to understand and create distinction of map layers. It is recommended to try out various symbol combinations for all layers. This is exceptionally important at the end when analyzing our results and displaying them for an audience.

  • Right click on the layer, go to Properties, select Symbology, and click OK once satisfied. Make sure to click Apply each time you change the symbology.
    SymbologyTab.png


Rule based classification

To visually separate the highways from other roads, this can be done through symbology as well.

  • By going to the Symbology tab again, hover over the top drop down menu and select Rule-Based. Double click the row, type 'Highway' in Label, and then click the expression icon.
    Rulebased .png


  • Under the expression tab, enter "SUBCLASS" = 'Highway' by either manually typing it or under Field and Values, double click on SUBCLASS, double click =, type 'Highway', and then select OK.
    RuleBased Highways.png


  • Repeat this for other roads by clicking + icon to add a new rule, type 'Other Roads' in Label, and select Else as it's the other condition. In the Layers pane it should look something like this:
    RoadSeperation.png

Analysis

Once you are done setting up the environment, we can now start with the main purpose of the tutorial which is exploring two classification tools, DBSCAN and K-means, coupled with a boundary tool, Concave hull. Once this is done we will have a good picture on how we can proceed in assigning areas to investigate collisions in Ottawa.

  • To find the necessary tools, under the Processing tab select Toolbox, search or scroll to find both clustering methods within Vector analysis or Concave hull (by layer) within Vector geometry.
    FindingClusteringTools.png


DBSCAN clustering

Density based clustering in QGIS will help us determine and locate clusters of accidents based on the point densities within our collisions dataset. This requires a minimum cluster size (# of points) and a maximum distance between these points. A scenario where this would be practical is if City of Ottawa wanted to know how many patrollers need to be allocated for this specific location.

Running the tool

  • Select DBSCAN clustering, add projected collisions layer into Input layer, select a desired value for Minimum cluster size and Maximum distance between cluster points - we are working in meters so it is best to stay in this. Then label and specify the path and press Run.
    • Tips: For min size, it is best to use a value greater than the dimension you are working in. For example, this is 2D so I would like to set my value at 3 or higher. Max distance depends fully on the input data and logical sense associated with it. It is recommended to attempt many combinations of each parameter to see what yields the best result for your objective.

Based on this scenario, the parameter combination which captured both rural and urban clusters were a min size of 4 and distance of 500m.

DBSCAN rural 4min500m.png

Visualizing results

You will now have a new vector layer called "Clusters" with a table entry named "CLUSTER_ID". To visually differentiate these clusters, use unique colours.

  • Right click Clusters layer, go to "Symbology", select Categorized in the drop down menu of classification.
  • In Value, select "CLUSTER_ID", click on Classify to get a preview, scroll down and uncheck all other values to remove the noise (points that didn't fall into a cluster), and click Apply.
    DBSCAN cluster symbology.png

This is how it should appear:

DBSCAN rural output.png

Interpreting results

If the min cluster size was larger than 4, it would fail to capture rural areas. However, the output also poorly represents Ottawa's urban clusters in the downtown area by creating one mega cluster. The mega cluster contained over half of the dataset recorded collisions, thus providing no useful context. To account for this, re-running this tool on the mega cluster with same max density distance but large min clusters can help determine urban clusters.

  • Select one of the points in the mega cluster using the selection icon to find its cluster ID.
    FindingLargeClusterID.png
  • In the cluster layer attribute table, select Select by Expression and enter "CLUSTER_ID IS 'X'", then click Select Features, and export this selected feature layer.
    SelectingLargeCluster.png
  • Repeat the same DBSCAN steps for urban clusters, however change the min cluster to a much higher value to account for frequent collision occurrence in urban centers. For this scenario, a value of 5x higher (min = 20) was chosen.

Below are the new results. It is a better representation than just running the dataset on a one-fits-all parameter, however the core of Ottawa still shows weak context:

Final DBSCAN result.png

K-means clustering

K-means clustering creates clusters based on the number of clusters specified by the user. This is useful if you already have a set number of patrollers or divisions and want to geographically split their patrolling areas. Currently, Ottawa has 3 patrol operation divisions which would show a practical, simplistic use of K-means. Assuming there is a surplus in available resources and we can show ideal, complex scenarios, the tutorial separates Ottawa into 20 divisions.

  • Select K-means clustering tool, add the reprojected collisions layer into Input layer, change Number of clusters to 20, provide a name and source location under Clusters, and press Run.
    Kmeans 20 tool.png
  • To visualize the output, repeat the same steps as done in for the DBSCAN clustering Visualizing results.

Below are the results. K-means clustering excels at dividing urban centers, unlike DBSCAN, and overall is better at geographically spreading operation divisions equally:

Kmeans20 map result.png

Concave Hull

We now have our clusters and want to assign officials to a region with fixed boundaries. Concave Hull creates a vector polygon by tightly enclosing a boundary around the outer points within a cluster. By inputting either DBSCAN or K-means clustering output, we can make these polygons. Since both processes are identical, we will use K-means as it resulted in a preferred output.

With both K-means generating 20 clusters and DBSCAN with over 50, a streamlined method is more efficient than manually filtering shared cluster IDs to make individual polygons. Thus, we can apply batch processing to automate the Concave Hull tool.

Cluster splitting

Here we can split the individual cluster layer into multiple layers corresponding to its ID.

  • Under Processing Toolbox search for and select Split Vector Layer, and add the K-means layer to Input layer
  • Using the drop down tab for Unique ID field, select the attribute field for cluster IDs
  • In Output file type select shp., then name and select the file destination.
    Split IDs.png
  • Find the chosen directory and its shapefiles under Browser tab, then drag and drop them into the map pane.
    FindingSplitLayers.png

Running the tool

  • After opening Concave Hull (by layer) tool, select Run as Batch Process.
    CH Batch Button.png
  • Under Input layer double click Autofill... and choose Select Files... to add all 20 shp. layers.
    Select Batch Layer.png
  • Under Threshold insert "1" into the first row, then click Autofill... and Fill down. Do the same for Allow Holes and Multipart geometry into singleparts using "No".
    • A threshold of 1 produces smoother boundaries with less extreme vertices, to capture a larger extent of the region.
  • Under Concave Hull select the path and input a name.
  • Lastly, make sure to check Load layers on completion to see the output in the map pane and press Run.
    CH BatchProcess2.png

Interpreting Results

Below is the result of Concave Hull polygons overlayed by points for the specific cluster. This output provides us with a baseline on how to identify regions where patrolling operations should be designated.

Further analysis can be made to account for the cluster size of a specific region. For example, urban clusters (e.g. Centretown) have higher frequency of collisions occurring, thus assigning more patrol officers to these municipalities can account for this demand. Whereas rural locations such as North Gower can be under less surveillance.

CH Results blackpoints map.png

Conclusion

These clustering methods in QGIS are invaluable and relatively easy to use as shown in this tutorial. If there is a need for something to be divided, clustering performs well with vector data, making it an effective tool to use. Whether you are investigating collision trends or assigning door-to-door sales representative using house locations, these QGIS tools have incredible potential in terms of urban planning or any purposes where discrete divisions are required. While it may not be as advanced and intricate as other tools, such as ArcGIS Pro, QGIS is a great alternative to complete these tasks for its open source and easily accessible nature.

Future Work

There definitely is a lot of future work in terms of the potential of something like this. You can add more components and maybe include some other GIS tools such as network analysis to calculate the most cost affective ways to investigate those areas (less gas consumption is better and could save city money). Use interpolation to maybe predict possible incidences of accidents. Future work more from a QGIS perspective is to include more methods of clustering as well. There is a lot that can be done around vector points that contain location data but that would be more suited for a separate tutorial not to overwhelm a user.

References


Very good support video k-means clustering:

Retrieved from "https://dges.carleton.ca/CUOSGwiki/index.php?title=Exploring_Clustering_In_QGIS&oldid=24149"