Shapefile vs. GeoPackage vs. GML vs. GeoJSON

Metodic

Download standard dataset from https://github.com/OpenGeoLabs/shapefilemustdie/releases/download/v1.0.0/test-data.zip. The dataset contains layer buildings - polygon of Netherlands, originally downloaded from http://download.geofabrik.de/europe/netherlands.html. The Shapefile (unzipped, all files together) has about 3.3GB.
Convert input Shapefile to GeoPackage, GML and GeoJSON formats.

For all the formats, count time it takes to run ogrinfo on various types of tasks. Each task is running using Python timeit module.

                    from timeit import timeit
                    time = timeit(
                        """subprocess.run(["ogrinfo", "more", "params"],
                        stdout=subprocess.PIPE)""")

NOTE: Graphs are using only GeoPackage vs. Shapefile times. It was not possible to work with GeoJSON in size of 4GB. GML proofed to be working, but the times were generally very slow, it was meaning less to visualise them.

Repeat the measurment

You can use the script, which is part of the GitHub repository. You can either use standard dataset or apply it on yours.

Results

It was measured on an EC2 c5a.4xlarge instance (SSD, 32GB RAM, AMD® Epyc 7r32 16 cores)

File size [GB]

Read all data sequentially [ogrinfo ...]

Test time, till all features are read sequentially using simple ogrinfo command.

        ogrinfo gis.osm_buildings_a_free_1.shp gis.osm_buildings_a_free_1

Getting file metadata [ogrinfo -so ...]

Test time, till file metadata are displayed using ogrinfo command.

        ogrinfo -so \
            gis.osm_buildings_a_free_1.shp gis.osm_buildings_a_free_1

Find feature using FID [ogrinfo -fid ...]

Test time, till feature with given fid is identified using ogrinfo command.

        ogrinfo -fid 63347 \
            gis.osm_buildings_a_free_1.shp gis.osm_buildings_a_free_1

The area of interest [ogrinfo -spat ...]

Test time, till feature with given spat condition is identified using ogrinfo command.

        ogrinfo -spat 5.421254 52.129629 5.421254 52.129629 \
            gis.osm_buildings_a_free_1.shp gis.osm_buildings_a_free_1

Find feature using where condition [ogrinfo -where ...]

Test time, till feature with given where condition is identified using ogrinfo command.

        ogrinfo -where "name='OBS De Hobbit'" \
            gis.osm_buildings_a_free_1.shp gis.osm_buildings_a_free_1

Last modification: 2017-10-22
Initially created by: Jachym Cepicky, OpenGeoLabs s.r.o.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
Contribute: On GitHub