Multiple conflicting time valid records can be inserted

[DanBuchan]

Somehow when multiple jobs run the database allow writes of multiple, time conflicting, records. Even though all read and write functions on the database are set to atomic. That is a two records for a given md5 can be inserted on the same day an dget the same expiry date

At a guess this is because the code running on each blast client is not aware of the other. But I thought atomicity was handled on the postgres instance. hmmm.

[DanBuchan]

Here is a set of bad data the triggers this issue
bad_data.csv.gz

[DanBuchan]

Also the psiblast running client scripts should probably fail over to running the psiblast (and not try and insert to the cache) and then send a warning to the psipred-alerts slack channel

[DanBuchan]

Probably we shouldn't be in the business of trying to write and maintain out own caching software given how notoriously difficult a CS problem that is. The entire rest of the server uses standard frameworks for all the other hard problems we're not experts in (i.e. queuing), so I really have no idea why I thought I could fashion a caching server from scratch.

Probably the blast cache should be re-visited. I assume there exists a nice Journaling cache system build on top of redis or mongoDB that we could use drop in place instead and it wouldn't be too much work to replace my current solution.

Configuring it to recocrd the information we want and then updating the clients scripts should be all that's needed. Probably a week to do the work and some number of days to add the new cache to the DB server and switch off this django based version.