ckanext-dataspatial

A CKAN extension that provides geospatial awareness of datastore data.

A product of the Western Pennsylvania Regional Data Center.

Built upon the work fo the Natural History Museum in London.

Overview

This extension provides geospatial awareness of datastore data. This includes:

• Geospatial searches within datasets;
• Pushing GeoJSON data (properties and geometries) in to the datastore;
• Support for PostGIS;
• Support for tile servers that injest PostGIS data. (e.g. martin).
• todo: Spatial extent of datastore searches

Installation

Path variables used below:

• $INSTALL_FOLDER (i.e. where CKAN is installed), e.g. /usr/lib/ckan/default
• $CONFIG_FILE, e.g. /etc/ckan/default/development.ini

1. Clone the repository into the src folder:

cd $INSTALL_FOLDER/src
git clone https://github.com/NaturalHistoryMuseum/ckanext-dataspatial.git

2. Activate the virtual env:

. $INSTALL_FOLDER/bin/activate

3. Install the requirements from requirements.txt:

cd $INSTALL_FOLDER/src/ckanext-dataspatial
pip install -r requirements.txt

4. Run setup.py:

cd $INSTALL_FOLDER/src/ckanext-dataspatial
python setup.py develop

5. Add 'dataspatial' to the list of plugins in your $CONFIG_FILE:

ckan.plugins = ... dataspatial

Configuration

There are a number of options that can be specified in your .ini config file. They all have defaults set, so none are required.

Name	Description	Default
dataspatial.postgis.field	WGS data field in the PostGIS database	_geom
dataspatial.postgis.mercator_field	Mercator field in the PostGIS database	_geom_webmercator

Further Setup

PostGIS

To use this extension, your PostgreSQL database must have PostGIS support.

1. Install the correct version of PostGIS for your version of PostgreSQL: https://postgis.net/documentation/getting_started/

2. You will then need to create PostGIS columns on your resources. Invoking the command below will create the two columns named above (dataspatial.postgis.field and dataspatial.postgis.mercator_field) on table $RESOURCE_ID. One represents the WGS (World Geodetic System) data, and one uses the web mercator projection, which is useful for generating maps.

   ckan dataspatial create-columns $RESOURCE_ID -c $CONFIG_FILE

Usage

Geospatial metadata

GeoJSON files can be parsed without any extra metadata.

To parse tabular files, you must update the resources extra fields.

The file can't be parsed unless either dataspatial_longitude_field AND dataspatial_latitude_field are provided OR dataspatial_wkt_field is provided.

Writable Fields

Field	Description
dataspatial_longitude_field	Name of field that contains longitude data
dataspatial_latitude_field	Name of field that contains latitude data
dataspatial_wkt_field	Name of field that contains Well-Known Text data
dataspatial_fields_definition	Optional, Only used with GeoJSON resources. Must be a valid Fields json object. Used to provide field types when loading a GeoJSON into the datastore.'

Read-only fields

Field	Description
dataspatial_active	is true if resource has been georeferenced
dataspatial_status	status of georeferencing job
dataspatial_last_geom_updated	timestamp of last time georeferencing was conducted

Actions

dataspatial_submit

Submit resource qeoreferencing job to be processed by CKAN worker.

curl -X POST https://data.wprdc.org/api/action/dataspatial_submit \
   -H 'Content-Type: application/json' \
   -H 'Authorization: <API_KEY>' \
   -d '{"resource_id": "<RESOURCE_ID>"}'

from ckan.plugins import toolkit

toolkit.get_action('dataspatial_submit')(
    context,
    {
        'resource_id': '<RESOURCE_ID>',
    }
)

dataspatial_status

Get georeferencing status of a resource with the following data:

Field	Description
job_id	ID of CKAN worker job
status	Status of the worker job
last_updated	Timestamp of last time the status was updated
rows_completed	Number of rows parsed
notes	Description of current status

curl -X GET https://data.wprdc.org/api/action/dataspatial_status?resource_id=<RESOURCE_ID>

from ckan.plugins import toolkit

status = toolkit.get_action('dataspatial_status')(
    context,
    {
        'resource_id': '<RESOURCE_ID>',
    }
)

datastore_search

Searching by geospatial fields involves passing a custom filter to datastore_search. The filter _tmgeom contains a WKT (Well-Known Text) string representing the area to be searched ( currently, only the types POLYGON or MULTIPOLYGON will work). e.g.:

from ckan.plugins import toolkit

search_params = {
    'resource_id': '<RESOURCE_ID>',
    'filters': '_tmgeom:POLYGON(36 114, 36 115, 37 115, 37 114, 36 114)'
}
search = toolkit.get_action(u'datastore_search')(context, search_params)

CLI

dataspatial

1. create-columns: create the PostGIS columns on the $RESOURCE_ID table.

   ckan dataspatial create-columns $RESOURCE_ID --geom-type=$GEOM_TYPE -c $CONFIG_FILE

2. create-index: create index for PostGIS columns on the $RESOURCE_ID table.

   ckan dataspatial create-index $RESOURCE_ID -c $CONFIG_FILE

3. populate-columns: populate the PostGIS columns from the given lat & long fields. Equivalent to the update_geom_columns() action.

   ckan dataspatial populate-columns $RESOURCE_ID -l $LATITUDE_COLUMN -g $LONGITUDE_COLUMN -c $CONFIG_FILE

Testing

tests coming soon

Acknowledgements

Based on ckanext-dataspatial created by the Natural History Museum in London, UK.