Healthcheck scenarios for diagnosis (proof of concept)

[benoitbryon]

Scenarios make it possible to run further checks in case of a failure.

As an example, if http://example.com does not return HTTP 200 within 1 second, run additional checks to diagnose the issue, such as does example.com responds to ping?

[benoitbryon]

Not sure nose's test generators do the trick...

@hospital.healthcheck
def test_example_com():
    yield check_http_200
    yield check_ping

def check_http_200():
    hospital.assert_http_response('http://example.com', status_code=200, timeout=1)

def check_ping():
    hospital.assert_ping('example.com')

In the healthcheck (test_example_com) function, I do not know how to do the switch: I would like to switch on the test result, not on the callable. Whatever the result of check_http_200, check_ping will be executed too.

[benoitbryon]

Perhaps using a TestSuite with implementation of either run() or __iter__() could do the trick.

[benoitbryon]

Note: the following is not what we need:

def test_something():
    hospital.assert_http_response('http://example.com')
    hospital.assert_ping('example.com')

If the first assertion succeeds, the second one is also triggered.

=> I want the second assertion to be triggered only if the first one failed.

[benoitbryon]

A naive implementation could be:

logger = logging.getLogger('healthchecks')

@hospital.healthcheck
def test_example_com():
    try:
        hospital.assert_http_response('http://example.com')
    except AssertionError as e:
        logger.error(e)  # Report error! But do not stop execution.
        hospital.assert_ping('example.com')

The implementation above implements a scenario, but there are some drawbacks:

• in the report there will be no relationship between the two errors. I'd like to see something like: "failed http response, let's check more... failed ping too".
• one error is reported with logging, and the other is reported with AssertionError. The way they are reported should be consistent.

[benoitbryon]

Quickly tried to declare a subclass of unittest.TestSuite in a healthcheck file. It is not collected by nose :(

I.e. something like this does not trigger the AssertionError (else I would have tried to run a test case, and another test case depending on the result).

class PredictableTestSuite(unittest.TestSuite):
    def run(self, result):
        assert False

[benoitbryon]

Another idea about making scenarios is to use test attributes: have a set of healtchecks with additional "is_smoketest" attribute.

In order to get the service status, I run only smoketests => if everything passed, then there is no need to run more checks.
Else, if some smoketest failed, I can run additional checks. Either all healthchecks, either a subset (as an example, the healthchecks in same TestCase or same module than failing smoketest).

import hospital

@hospital.smoketest
def test_http_200():
    hospital.assert_http_response('http://example.com', status_code=200, timeout=1)

@hospital.healthcheck
def check_ping():
    hospital.assert_ping('example.com')

[benoitbryon]

Let's implement #35 for version 0.4 and delay this one.