ECDP not working correctly

[GoogleCodeExporter]

Hi:
  I'm not seeing all the runs in ECDP and some of the ones that are displaying are not showing proper designations (i.e. analyses are done and not showing).  Also seems like a good time to put a notice that the system is having some issues, as opposed to the default launch screen saying there are none.  Thanks!

Original issue reported on code.google.com by cmnico...@gmail.com on 5 Sep 2014 at 5:54

[GoogleCodeExporter]

1) maybe we need another update 
or
2) restore to last good copy and then update
or
3)
full rebuild.

Original comment by zack...@gmail.com on 5 Sep 2014 at 5:59

[GoogleCodeExporter]

I am going to try approach 2) first.
I am going to put a notice saying : 
"We are working on reported issues. Sorry for inconvenience."

Original comment by natalia....@gmail.com on 5 Sep 2014 at 6:53

[GoogleCodeExporter]

There were 3 insertQC.pl processes running, from previous days probably. Killed 
them, restored db to August 25 copy and started update

Original comment by natalia....@gmail.com on 5 Sep 2014 at 7:42

[GoogleCodeExporter]

Yeah, That was probably the problem.

We should check if another process is running and die if true, to prevent this

Original comment by zack...@gmail.com on 5 Sep 2014 at 7:51

[GoogleCodeExporter]

How about making sequencingDevel_Everyday.sh look like this:

#!/bin/sh
p=`ps -ef | grep insertQC.pl | awk END'{print NR}'`;
if [ $p -eq 1 ] 
then 

./insertQC.pl 4

fi

Original comment by natalia....@gmail.com on 5 Sep 2014 at 8:37

[GoogleCodeExporter]

Storage is still slow. It takes 3 sec for ls /storage to return

Original comment by natalia....@gmail.com on 5 Sep 2014 at 8:42

[GoogleCodeExporter]

unfortunately this is somewhat expected since a lot of disk operations are 
happening on the head node right now. 

Original comment by zack...@gmail.com on 5 Sep 2014 at 9:17

[GoogleCodeExporter]

I started the update at 12:40 and the process is hanging since 18:22. I am 
curious when storage is going to be back to normal. I disabled the update since 
the current one is hanging.

Original comment by natalia....@gmail.com on 6 Sep 2014 at 3:42

[GoogleCodeExporter]

is it completely hung or just very slow, there is a big difference between the 
two.

you can ssh into hpc-laird and see the load. it looks as if moiz is backing up 
hundreds of gigs.

Original comment by zack...@gmail.com on 6 Sep 2014 at 3:46

[GoogleCodeExporter]

Issue 807 has been merged into this issue.

Original comment by zack...@gmail.com on 6 Sep 2014 at 3:46

[GoogleCodeExporter]

also, have you checked the load on the other VM. that is the storage gateway. 
if there are many stalled rsyncs happening there, then it is definitely part of 
the problem.

Original comment by zack...@gmail.com on 6 Sep 2014 at 3:49

[GoogleCodeExporter]

nevermind, I checked and there were some rsyncs going, i killed them

Original comment by zack...@gmail.com on 6 Sep 2014 at 3:51