CSX: major BS bug with suspenders

[cmazzoli]

After the on the fly fix by J. Lhermitte and S. Campbell yesterday evening, this morning we have been hit again by a beam dump. As a result we got the following.
I don't know if the fact that I had to manually close the photon shutter to perform a rescue intervention on a piece of hardware across the beam re-delivery can have caused additional problems, how ever it is worth investigating..

|       312 | 10:13:10.3 |   8.600000 |      -0.910 |   2513.0064 |               5263.0135 |           2647457 |          15982857 |                                         
|       313 | 10:13:15.6 |   8.600000 |      -0.920 |   4825.0062 |              10071.0197 |           2615154 |          15942823 |                                         
Suspending....To get prompt hit Ctrl-C twice to pause.                                                                                                                        
Suspension occurred at 2018-03-30 10:13:20.
Justification for this suspension:
Signal XF:23ID-SR{}I-I = 15.332104004312395 is below 250
Suspending....To get prompt hit Ctrl-C twice to pause.                                                                                                                        
Suspension occurred at 2018-03-30 10:18:40.                                                                                                                                   
Justification for this suspension:
Signal XF:23ID-PPS{Sh:FE}Pos-Sts is high
Suspending....To get prompt hit Ctrl-C twice to pause.
Suspension occurred at 2018-03-30 11:57:26.
Justification for this suspension:
Signal XF:23IDA-PPS:1{PSh}Pos-Sts is high
Suspender SuspendFloor(EpicsSignal(read_pv='XF:23ID-SR{}I-I', name='XF:23ID-SR{}I-I', value=251.5404206931987, timestamp=1522426513.252614, tolerance=0.01, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:23ID-SR{}I-I', limits=False, put_complete=False), sleep=180, pre_plan=None, post_plan=None,tripped_message=) reports a return to nominal conditions. Will sleep for 180 seconds and then release suspension at 2018-03-30 12:18:13.
Suspender SuspendBoolHigh(EpicsSignal(read_pv='XF:23ID-PPS{Sh:FE}Pos-Sts', name='XF:23ID-PPS{Sh:FE}Pos-Sts', value=0, timestamp=1522426609.871464, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:23ID-PPS{Sh:FE}Pos-Sts', limits=False, put_complete=False), sleep=600, pre_plan=None, post_plan=None,tripped_message=) reports a return to nominal conditions. Will sleep for 600 seconds and then release suspension at 2018-03-30 12:26:49.
Suspender SuspendBoolHigh(EpicsSignal(read_pv='XF:23IDA-PPS:1{PSh}Pos-Sts', name='XF:23IDA-PPS:1{PSh}Pos-Sts', value=0, timestamp=1522427416.073126, pv_kw={}, auto_monitor=False, string=False, write_pv='XF:23IDA-PPS:1{PSh}Pos-Sts', limits=False, put_complete=False), sleep=300, pre_plan=None, post_plan=None,tripped_message=) reports a return to nominal conditions. Will sleep for 300 seconds and then release suspension at 2018-03-30 12:35:16.
+-----------+------------+------------+-------------+-------------+-------------------------+-------------------+-------------------+
generator grid_scan ['0b822233'] (scan num: 99138)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~/Beamline/ScienceComm/2018_03_ZP/20180329_night.py in <module>()
     23 RE(grid_scan(dets,sx,7.8,8.8,31,sy,-0.25,-1.25,31,0))
     24 # angular offset
---> 25 RE(mvr(theta,-6.0))
     26 RE(grid_scan(dets,sx,8.5,8.8,31,sy,-0.9,-1.2,31,0))
     27 

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in __call__(self, *args, **metadata_kw)
    668                     # it (unless it is a canceled error)
    669                     if exc is not None:
--> 670                         raise exc
    671 
    672             if self._interrupted:

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1117             self.log.error("Run aborted")
   1118             self.log.error("%r", err)
-> 1119             raise err
   1120         finally:
   1121             # Some done_callbacks may still be alive in other threads.

/opt/conda_envs/collection-2018-1.0.1/lib/python3.6/site-packages/bluesky/run_engine.py in _run(self)
   1014                     # The normal case of clean operation
   1015                     else:
-> 1016                         resp = self._response_stack.pop()
   1017                         try:
   1018                             msg = self._plan_stack[-1].send(resp)

IndexError: pop from an empty deque

In [65]: 

I don't know if the fact that I had to manually close the photon shutter to perform a rescue intervention on a piece of hardware across the beam re-delivery can have caused additional problems, how ever it is worth investigating..

[tacaswell]

@cmazzoli I see another bug in here in that 20180329_night.py is calling RE multiple times!

[jrmlhermitte]

@cmazzoli to follow up with this. @mrakitin @tacaswell and I added the fix that @tacaswell made.
We could not test it as the bug occurs when more than one suspender kicks in.
This can be tested by closing the front end shutter, which will trigger the front end and endstation suspender. If you have time, I would suggest giving that a try. (we obviously could not trigger this shutter since CSX2 is also using beam. We also didn't have time to test a new suspender)

Another option is that we can add another suspender in on one of your devices you're not using, and change its state. We can meet to follow up with this.

Which do you prefer? Testing yourself with FE shutter, or meeting in person later and setting up a suspender. For example, for a device whose only values are 0 or 1:

fe_shut_suspender = SuspendBoolHigh(EpicsSignal('XF:23ID-PPS{Sh:FE}Pos-Sts'), sleep=10*60)

where fe_shut_suspender should be a different name and the pv replaced with the PV.

[cmazzoli]

Hi guys, Either is fine with me. To access the FE we need no radiation in the ring as it would perturb IOS beamline.. About the many REs.. Tom you are more than welcome at 3am on the beamline to help me with plans :) . Please let me know what you prefer to do. Claudio

…

On Thu, Apr 5, 2018, 09:55 Julien Lhermitte ***@***.***> wrote: @cmazzoli <https://github.com/cmazzoli> to follow up with this. @mrakitin <https://github.com/mrakitin> @tacaswell <https://github.com/tacaswell> and I added the fix that @tacaswell <https://github.com/tacaswell> made. We could not test it as the bug occurs when more than one suspender kicks in. This can be tested by closing the front end shutter, which will trigger the front end and endstation suspender. If you have time, I would suggest giving that a try. Another option is that we can add another suspender in on one of your devices you're not using, and change its state. We can meet to follow up with this. Which do you prefer? Testing yourself with FE shutter, or meeting in person later and setting up a suspender. For example, for a device whose only values are 0 or 1: fe_shut_suspender = SuspendBoolHigh(EpicsSignal('XF:23ID-PPS{Sh:FE}Pos-Sts'), sleep=10*60) where fe_shut_suspender should be a different name and the pv replaced with the PV. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#192 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKjlFlfH97Zuh0UqbjPvSrbAkmNwpt2uks5tliItgaJpZM4TB43i> .