Calculating historical agricultural land differences using QGIS

Introduction One issue when working with large data sets is the massive resources required to render the files. This is not so bad when you have a computer powerful enough to handle them, or you absolutely need to have data that detailed, but there are problems in other situations. I found myself in such a situation about a year ago. I was attempting to graphically represent agricultural land attrition in Southern Ontario, from 1983 – 2017. The data I had downloaded were incredibly detailed, which is perfect if you wanted to grab a subset and analyze it on a micro level, but not so great if you want to analyze at a macro level. While I was able to complete my analysis, it was frustratingly time consuming. Every map movement caused a render lag, and any use of analysis tools resulted in a long wait. This leads me to now. While I was not able to think up a way to do it back then, I figured now might be the time to try. This led me to a feature in QGIS called ‘dissolve’. While this is a general tool in many GIS software packages, I decided to see if I could improve my past project, using QGIS Software: QGIS LTR 3.44 Solothurn (tested on Windows 11). Updated:October 6th 2026 (original tutorial 2019 - edited 2020) The original tutorial (2019/2020) has been revised to repair invalid geometry instead of discarding it, to correct how dissolve is described, to measure area in an equal-area projection, and to note that the two datasets differ in coverage and classification. Benchmark Info This revision of the tutorial was run on a relatively powerful PC so if using a laptop or older machine processing times can be higher. Cloud Computing Cloud computing is a fairly recent solution to having a large dataset that can take a long time to process, or when you cannot use the dissolve tool described below, or if your computer is not particularly powerful, because it lets you finish the work in a reasonalble amount of time. There are several ways to do this, depending on the software you want to use. Large online companies such as Amazon and Google rent server spacem so if your data is too large to process on your own computer you can run the work on their servers and keep doing other work in the meantime. Cloud services cost money and need set-up time, so try the optimizations in this tutorial (dropping unneeded fields, repairing geometry, dissolving, saving to GeoPackage) first. Data Two files are needed for this project: the 1983 Agricultural Resource Inventory, and the most current Agricultural Resource Inventory. The source data are constantly being updated, so I have provided a link to the homepage of each file, rather than a direct download link. Once on the page, select 'open', and the data will begin downloading. For the 1983 data, please visit: https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-1983 For the current data, please visit: https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-final Due to issues surrounding QGIS' geodatabase handling ability, I have provided a link to the gdb data converted to a shp file format, below. For those with ArcGIS access, the original file in the link above can be opened in ArcMap and exported as a shapefile that will work with QGIS. Check your data first Before processing, check each layer: CRS and extent right-click the layer > Properties > Information. Add a base map (browser panel -> XYZ Tiles -> OpenStreetMap) to confirm the layers line up with real roads and fields. A layer in the wrong place (for example if the data ends up in the middle of the ocean) must have a missing or wrong CRS. Make sure all the files that came with a shapefile, including the .prj, are in the same folder. If the CRS is wrong, find the correct one from the metadata, apply it with Layer > Set CRS of layer, and save it with the Assign projection tool. Do not use Reproject to fixe a wrong label. For the 1983 data and the 2017 data the CRS should be NAD 83 / Ontario MNR Lambert. The project CRS should be Canada Albers Equal Area Conic, this can be selected in project properties. Coverage: the two datasets do not cover the same area or use the same classes (see Limitations Section). Zoom out and see where both layers have data. For a fast run, it is viable to create a study area where both datasets overlap:Processing Toolbox > Create layer from extent > Draw on Canvas, and save it as a GeoPackage. Tutorial Instructions Step 1: Identifying the Problem In this image we can see the problem. The data are so dense that changes between the two cannot be seen. This data is really meant for querying, not so much for display. Figure 1. Problem with data display Looking at this, it isn’t hard to see why we’re having the difficulties we are. There are 555,000 individual features in six fields, in the 2017 data, alone. We need to pare that down. All we want to keep is the ‘Shape_area’, which is in meters squared. This will allow us to calculate a final area. Figure 2. Massive amounts of attribute data Tip: large layers make every step slow. For a faster first pass, work on a study area whee both datasets have data (see "Check your data first"), then repeat on the full data. Step 2: Deleting Fields By deleting unnecessary fields, we can hopefully reduce the size and complexity of the data. Figure 3. How to access editor for field deletion Ensure that the editing tool is toggled on (highlighted in red), then open the new files attribute table, and delete the unnecessary fields. Figure 4. Fields to be deleted Everything except ‘Shape_Area’ should be removed. This can take a while. Once completed, save layer edits by selecting the icon on the right of the editing tool. Editing can be toggled off at this point as well. Alternative (potentially faster method): in the Processing Toolbox, search for Drop field(s). Choose the layer, click next the ... next to "Fields to drop", select everything except the area field (Shape_Area in the 2017 layer and HECTARES in the 1983 layer) and save the outpute as a Geopackage to be used for the analysis. [Manually deleting time: ~1 minute, Drop field(s) time: ~15 seconds ] Step 3: Check Data Validity and Fix the Data This is a very important step. Unlike ArcGIS, if there are any errors or invalid data, QGIS will quit whatever process is running if a feature has invalid geometry, for example Feature (8337) from "AgRI -- ARI_1983" has invalid geometry. Fix the geometry or change the 'Invalid features filtering' option." We need to be sure that there are no errors in our shapefile geometry. To do this, we need to use the Check Validity tool. Figure 5. Navigate to the check validity tool This is the first step. If there are errors in the data, or some of it is invalid, QGIS will not be able to process it. The ‘Check Validity’ tool will allow us see data that needs fixing, and ensure that processing proceeds smoothly. Figure 6. Select the file, and leave all options as default Once the validity checker has completed its operation, there should be three files. Valid output, Invalid output, and error. It also reports how many features fall into each. DO NOT simply dicard the invalid features as removing them deletes data from your totals, which produces an untrue result. In one test, removing invalid features from [ARI_1983] reduced the total features by 363. Instead, repair them: in the Processing Toolbox run Fix geometries on the layer and save the output to a GeoPackage using Save to GeoPackage. Use the fixed layer for the remaining steps. Now the feature count before and after should match (feature count for ARI_1983 should be 139277 before and after). Tip: to work on a smaller area, select the features around your study area first (select features by rectangle), then run the check validity and fix geometries with "selected features only ticked". Then cut the fixed layer to the study area with the "Clip" tool. Figure 7. Fixed geometries processing tool screenshot When the new file has been saved, rename the fixed geometries output that QGIS created to ___ fixed and make it permenant, you can keep or discard the old ARI_2017/ARI_1983 files, and open the newly created file from where you have saved it. Be sure to duplicate this process so you have ARI_1983_Fixed and ARI_2017_Fixed. Step 4: Dissolve Polygons The dissolve tool will allow us to merge all the individual polygons into one. This should make it much less resource intensive to render the data. This will make it easy to compare with the field calculator later on. Fair warning, this process for both files was somehow the longest, and most varied in terms of time. Each file will need anywhere from 15 minutes to more than five hours to process. This appears to be based on the number of attributes, as well as relative density of the data. With closer together 'dense' data being faster to process than more spread out 'less dense' data. think of this tool being like an addition tool for your polygons it will add all of the surface area into one so that when we move it around and manipulate it the computer doesn't have to do this with thousands of polygons it just has to do this with one polygon which is much simpler. We can use this tool for this because we are using area as our variable some variables would need to be kept in their separate polygons, so make sure to check that your data is compatible before using this tool. Figure 9. Navigate to the dissolve tool Once completed, we can then finish the processing by dissolving the individual breaks, and creating a seamless surface. Figure 10. Settings for the dissolve tool Open the dissolve tool, ensure ARI_2017_FIXED or ARI_1983_FIXED is selected, and run the tool. Save the new file to make it permanent (geopackage), and leave all other options the same. This process can take a while. Because the inputs had fixed geometries and the dissolve being run on newer QGIS software + more powerful computer with better RAM and CPU components the processing went very fast... In this test the dissolve took 1 minute 27 seconds for ARI_1983 (139000 features) and it took 1 hour 26 minutes for ARI_2017 on the full fixed datasets (see the Benchmark section(s)). Dissolve is only suitable here because the question concerns total area. If you need to keep the attributes of individual polygons, do not dissolve. Step 5: Field Calculator Area should be measured in an equal-area projection. Reproject each dissolved layer with the Processing Toolbox tool "Reproject layer", setting the target CRS to "Canada Albers Equal Area Conic", and save the result. (This is quick because each dissolved layer contains only one feature.) Now that processing is complete, we need to calculate the area of each file. This will give us a number, that can then be converted into percent change. When QGIS dissolves the ARI_2017_Fixed file, it keeps only one area entry. This means that area must be calculated from ARI_2017_fixed, and manually added to the new dissolved layer (ARI_2017_dis). To access the Field Calculator, open the attribute table, toggle on editing and navigate over to the abacus icon. It is one to the right of the delete field icon highlighted in figure 3. Figure 11. Field Calculator for individual area calculations First, we have to sum the 'area' field manually using SUM(Shape_area)/10000. This is because Shape_area is in meters squared, and we need to convert to hectares, which is what the 1983 data is in. For calculating the total area in ARI_1983, the code will be SUM(HECTARES). Once we have a single value for both fields, navigate to the dissolve file, open the attribute table, enable the editor, and manually update the value for both dissolved files. Independently, of course. Figure 12. Update table value to include total area in hectares Above is an example of the updated value in ARI_1983_dis. By double clicking on the attribute, we can update it manually with the correct value. % change = (A_2017 - A_1983) / A_1983 * 100 Step 6: Difference This allows us to generate a shapefile of the differences between two files, making it extremely easy to visually represent what has changed. Figure 13. Difference tool navigation To access the difference tool, navigate over to Vector > Geoprocessing tools. Figure 14. Difference tool Settings Set the larger layer as the input, and the smaller layer as the overlay. In this case, ARI_1983_dis is the input, and ARI_2017_dis is the overlay. Ensure the file is saved in a known location (Geopackage). Final Output Once all processing is completed, we can see the output. Dissolve worked much better for ARI_1983 (Orange), than it did for ARI_2017 (Black but supposed to be Green). When zooming in, borders can be clearly seen around the image which is probably because each piece of agricultural land in ARI_2017 was delineated much better, providing proper breaks between each field. This would also explain why there are many more attributes present in ARI_2017. Figure 15. Final map output after all processing Using the field calculator (sort of) we are able to calculate the percent difference between 1983 and 2017. The difference is highlighted in red. 66.8% drop from 1983 to 2017. Figure 16. Percent difference between 1983 & 2017

Benchmark

Times and file sizes for [full data with fixed geometries] on [CPU: Intel i5 14400f, RAM: 16GB DDR5]

ARI 1983 Geodatabase File Size = 413 MB, ARI 1983 Fixed Geometries File Size = 10.364 MB || ARI 2017 Geodatabase File Size = 200 MB, ARI 2017 Fixed Geometries File Size 198.864 MB

ARI 1983 feature count = 139000, Dissolve time = 1 minute 27 seconds || ARI 2017 feature count = 550000, Dissolve time = 1 hour 26 minutes 47 seconds

Dissolve is a Valid tool to make large datasets easier to work with.

Conclusion

When you are working with large datasets that have lots of polygons you can use the dissolve tool to make the data more manageable, however we need to be careful that we still keep the data accurate. If your computer is not powerful enough to do the calculations involved in this tool we also can use cloud computing to get this done in a more reasonable time. And in terms of the data we used in the example it turned out that using this method we were more easily able to tell that the amount of agricultural land shown in each dataset declined by around 66.8% from 1983 to 2017 (not tested with fixed geometries but likely similar).

Limitations

• The 1983 and 2017 datasets are not the same. According to the dataset description ARI-Final (the 2017 one) differs from ARI1983 in coverage area and in classification: ARI 1983 gives specific crops and field use for Southern Ontario, while ARI-Final broadly classifies land use (farms, fields, fences, and roughland) for project areas across ontario.
• Dissolve removes all individual polygon attributes.
• Results depend on the study area and the geometry repairs used, whether it is decided to use fix geometries, and if a subset area is used such as a county/jurisdiction or if the full Ontario datasets are used.

References

• QGIS Documentation. https://docs.qgis.org/
• Land Information Ontario GeoHub. Agricultural Resource Inventory 1983. https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-1983
• Land Information Ontario GeoHub. Agricultural Resource Inventory - Final. https://geohub.lio.gov.on.ca/datasets/agricultural-resource-inventory-final