When i delete a rule it does not stop the alerting from that rule . The only way is to restart a elastalert I think that was the old problem but was eventually solved on elastalert tree