Calculating historical agricultural land differences using QGIS

Introduction

One issue when working with large sized data sets is the massive resources required to render the files. This is not so bad when you have a computer powerful enough to handle it, or you absolutely need to have data that detailed, but it is when you do not. I found myself in that situation about a year ago. I was attempting to graphically represent agricultural land attrition in Southern Ontario, from 1983 – 2017. The data I had downloaded was incredibly detailed, perfect for if you wanted to grab a subset and analyze it on a micro level. Not so great when you are trying to analyze on a macro level. While I was able to complete my analysis, it was frustratingly time consuming. Every map movement caused a render lag, and any use of analysis tools resulted in a long wait. This leads me to now. While I was not able to think up a way to do it back then, I figured now might be the time to try. This lead me to a feature in QGIS called ‘dissolve’. While it is also available in ArcPro, that is not the purpose of this assignment. I decided to see if I recreate my project, using QGIS instead of Arc, and perhaps improve upon it a little.

Data

Two files are needed for thhis project, the 1983 Agricultural Resource Inventory, and the most current Agricultural Resource Inventory. The source is constantly being updated, so I have provided a link to the homepage of each file, rather than a direct download link. Once on the page, select 'open', and the data will begin downloading.

For the 1983 data, please visit: https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-1983

For the current data, please visit: https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-final

• Due to issues surrounding QGIS' geodatabase handling ability, I have provided a link to the gdb data converted to a shp file format, below. For those with ArcGIS access, the original file in the link above can be opened in ArcMap and exported as a shapefile that will work with QGIS.

Tutorial Instructions

Step 1: Identifying the Problem

In this image we can see the problem. The data is so dense that changes between the two cannot be seen. This data is really meant for querying, not so much for display.

Figure 1. Problem with data display

Looking at this, it isn’t hard to see why we’re having the difficulties we are. There are 555,000 individual features in six fields, in the 2017 data, alone. We need to pare that down. All we want to keep is the ‘Shape_area’, which is in meters squared. This will allow us to calculate a final area.

Figure 2. Massive amounts of attribute data

Step 2: Deleting Fields

By deleting unnecessary fields, we can hopefully reduce the size and complexity of the data.

Figure 3. How to access editor for field deletion

Ensure that the editing tool is toggled on (highlighted in red), then open the new files attribute table, and delete the unnecessary fields.

Figure 4. Fields to be deleted

Everything except ‘Shape_Area’ should be removed. This can take a while. Once completed, save layer edits by selecting the icon on the right of the editing tool. Editing can be toggled off at this point as well.

Step 3: Check Data Validity

This is a very important step. Unlike ArcGIS, if there is any errors or invalid data, QGIS will quit whatever process is running.

Now that we have deleted all unnecessary fields, we need to be sure that there are no errors in our shapefile geometry. To do this, we need to use the Check Validity tool.

Figure 5. Navigate to the check validity tool

This is the first step. If there are errors in the data, or some of it is invalid, QGIS will not be able to process it. The ‘Check Validity’ tool will allow us to remove any bad data, and ensure that processing proceeds smoothly.

Figure 6. Select the file, and leave all options as default

Figure 7. Validity Checker outputs

Once the validity checker has completed its operation, there should be three files. Valid output, Invalid output, and error. Remove the invalid and error, then right click on Valid_output and select Make Permanent…

Figure 8. New layer outputs

When the new file has been saved, remove Valid_output, as well as the old ARI_2017 file, and open the newly created file from where you have saved it. You should wind up with something like this. You may not see the new file in the QGIS browser. If this happens, simply hit the refresh button. Be sure to duplicate this process for ARI_1983.

Step 4: Dissolve Polygons

The dissolve tool will allow us to merge all the individual polygons into one. This should make it much less resource intensive to render the data. It will also merge the area for each polygon into one attribute. This will make it easy to compare with the field calculator later on. Fair warning, this process for both files was somehow the longest, and most varied in terms of time. Each file will need anywhere from 15 minutes to more than 60 to process. This appears to be based on the number of attributes, as well as relative density of the data. With closer together 'dense' data being faster to process than more spread out 'less dense' data.

Figure 9. Navigate to the check validity tool

Once completed, we can then finish the processing by dissolving the individual breaks, and creating a seamless surface.

Figure 10. Settings for the dissolve tool

Open the dissolve tool, ensure ARI_2017_chk is selected, and run the tool. Save the new file as a shapefile, and leave all other options the same. This process can take a while.

Step 5: Field Calculator

Now that processing is complete, we need to calculate the area of each file. This will give us a number, that can then be converted into percent change.

When QGIS dissolves the ARI_2017_chk file, it keeps only one area entry. This means that area must be calculated from ARI_2017_chk, and manually added to the new dissolved layer (ARI_2017_dis). To access the Field Calculator, open the attribute table, toggle on editing and navigate over to the abacus icon. It is one to the right of the delete field icon highlighted in figure 3.

Figure 11. Field Calculator for individual area calculations

First, we have to sum the 'area' field manually using SUM(Shape_area)/10000. This is because Shape_area is in meters squared, and we need to convert to hectares, which is what the 1983 data is in.

Once we have a single value for both fields, navigate to the dissolve file, open the attribute table, enable the editor, and manually update the value for both dissolved files. Independently, of course.

Figure 12. Update table value to include total are in hectares

Above is an example of the updated value in ARI_1983_dis. By double clicking on the attribute, we can update it manually with the correct value.

Step 6: Difference

This allows us to generate a shapefile of the differences between two files, making it extremely easy to visually represent what has changed.

Final Output

Once all processing is completed, we can see the output. Notice how it shows as a single colour, rather than many tiny polygons which show up as black. 1000px