README - HDX JOB PROCESSOR

Name

HDX JOB PROCESSOR, formerly known as "GISRestLayer" or "Gislayer"

Short description

This tool is evolving from just providing a way to asynchronously run the Geopreview process to a more general tool for asynchronously executing tasks for HDX (CKAN). For now it just deals with 2 tasks:

1. Geopreview Processing
2. Sending Analytics Events

The tools is based on Redis Queue

Geopreview Processing

This manages the transformation process of GIS data files ( geojson, kml, kmz, zipped shapefiles ) into a Postgres / PostGIS database. From there, the GIS data can be served by using the PGRestAPI server ( aka Spatial Server ). The gained advantages is that the PGRestAPI can serve:

• Mapbox Vector Tiles
• simplfied geometries depending on zoom level

How geopreview works

1. User uploads GIS data file on CKAN

2. HDX JOB PROCESSOR is notified about the new file

3. GISRestLayer then delegates all the import process tasks to Redis Queue ( and it’s workers ). The name of the queue geo_q . The worker code is part of this repository in: importapi.tasks.create_preview

4. The worker downloads the data

5. Uses ogr2ogr to import it in PostGIS

   • after this is finished PGRestAPI can already start serving tiles

6. Notifies CKAN that the task is complete. If successful, CKAN can show the user a preview of the data

Notes

• We're using a slightly modified fork of PGRestAPI https://bitbucket.org/agartner/hdx-pgrestapi It allows serving new layers without restarting the server

Running the redis queue worker for geopreview

The code below needs to be run in the folder where the python code is

./hdxrq.py worker --url redis://redis_ip:redis_port/1 --worker-ttl 600 geo_q

Please note that rqworker was replaced by ./hdxrq.py worker. More info can be found here

Sending Analytics Events

This manages the sending of data to analytics servers like Mixpanel or Google Analytics.

How analytics work

1. CKAN sends all the event metadata to HDX JOB PROCESSOR
2. HDX JOB PROCESSOR then creates tasks in Redis Queue in a queue called analytics_q .
3. Workers then send the information to analytics servers. The worker code is part of this repository in: analyticsapi.tasks.send_event

Running the redis queue worker for analytics

The code below needs to be run in the folder where the python code is

./hdxrq.py worker --url redis://redis_ip:redis_port/1 --worker-ttl 600 analytics_q

Please note that rqworker was replaced by ./hdxrq.py worker. More info can be found here

Generating events for the event bus

The "events" API is used for detecting changes in datasets and transforming them in events which are pushed to the event bus. Read more about this here

Scheduling feature (deprecated)

The HDX Job Processor has support for scheduling (though it's unused at the moment). To read more about it go here

Logging

To read more about how logging is configured, look here