Migrate sync process

[mosoriob]

According to @dnfeldman

there is supposed to be a background job that periodically parses dataset's metadata json and populates the relevant fields
which is not currently there...

Can you use a postgres trigger? It makes from me

The trigger can be specified to fire before the operation is attempted on a row (before constraints are checked and the INSERT, UPDATE, or DELETE is attempted); or after the operation has completed (after constraints are checked and the INSERT, UPDATE, or DELETE has completed); or instead of the operation (in the case of inserts, updates or deletes on a view). If the trigger fires before or instead of the event, the trigger can skip the operation for the current row, or change the row being inserted (for INSERT and UPDATE operations only). If the trigger fires after the event, all changes, including the effects of other triggers, are "visible" to the trigger.

[dnfeldman]

I don't think the trigger approach would work in this case. Reason being is that updating temporal and/or spatial information on a dataset level requires going through all resources, which is very expensive. It would mean waiting for ~5 min after any changes to resources. I think manual sync trigger is probably the easiest approach, especially if it is run as a cronjob

curl '<dcat_endpoint>/datasets/sync_datasets_metadata' -X 'POST' -H 'Connection: keep-alive'

(where the endpoint is data-catalog.mint.isi.edu)

Let me know what you think!

[mosoriob]

I see the problem.

Question?

What happened in the following use case?

1. The user adds a new dataset
2. The user adds 10 resources
3. The user search using the DataCatalog UI or MINT UI.

Should the user wait 5 minutes to use the datasets?

Deploy

I'm using the docker-compose.
Can you add a new container with the crontab?

FROM alpine:3.6

# copy crontabs for root user
COPY config/cronjobs /etc/crontabs/root

# start crond with log level 8 in foreground, output to stderr
CMD ["crond", "-f", "-d", "8"]

Where cronjobs is the file that contains your cronjobs, in this form:

* * * * * curl ${flask_app}/datasets/sync_datasets_metadata' -X 'POST' -H 'Connection: keep-alive'  2>&1
# remember to end this file with an empty new line

[dnfeldman]

The sync is there to update spatial and temporal coverages for a dataset. Everything else should be available immediately. So in the scenario you described, searching the dataset by its name should work right away, but searching by the specific spatial extent might return the newest results

As for adding a crontab, I just want to clarify: do you want to me to spin up a crontab container from the existing docker-compose.yml (https://github.com/mintproject/MINT-Data-Catalog/blob/master/docker-compose.yml)? Or do you have a separate crontab container that you use to manage all the running services on node1?

[mosoriob]

I prefer a crontab container because the services can be in the same place.