Is it possible to customize the output of WSGI app from within a testcase?

[benoitbryon]

See https://twitter.com/bartstroeken/status/467564038025932800

[benoitbryon]

What do you expect?
What would you like to customize? The error messages? The JSON structure? Something else?
Can you give an example?

[bartee]

Hi Benoit,

Thanks for responding, I thought about overwriting the TestResultResponse class, but there might be a better way.
I'd like to be able to annotate the details of a testcase, for some extra info like the host where that specific test was executed on - in case of an external service or server.

By adding that, you can interconnect different hosts running hospital tests for example.

{
    "status": "pass",
    "details": [
        {
            "test": "test_http_200 (healthchecks.exampletest.DocumentationHealthCheck)",
            "status": "pass",
            "domain": "http://api.nu.nl", 
        },
        {
            "test": "test_connection (healthchecks.mysqltest.MySQLTestCase)",
            "status": "pass",
            "host": "192.168.1.44"
        },
        {
            "test": "test_ping (healthchecks.redistest.RedisHealthCheck)",
            "status": "pass",
            "host": "192.168.1.55"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}

[benoitbryon]

There may be some improvements around the context data passed to the AssertionError.
As an example, at the moment, with a fail we get an additional "context" list:

{
    "status": "fail",
    "details": [
        {
            "test": "test_false (foo.bar)",
            "status": "fail",
            "context": "(<type 'exceptions.AssertionError'>, AssertionError(\"Failed to fetch URL http://localhost:8000/.\\nException was: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: / (Caused by <class 'socket.error'>: [Errno 111] Connection refused)\",), <traceback object at 0x7f34ac5bde18>)"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 0,
        "expected_failure": 0,
        "error": 0,
        "fail": 1,
        "total": 1,
        "unexpected_success": 0
    }
}

Note: the traceback in "context" is not really readable, see #72.
But here is one idea:

• on one side, we could improve the way hospital displays the "context"
• on the other side, developers could pass context data to assertions.

... but the fact is I'm not sure how we can easily do this. Using the assertion message? assert True is False, "Here some information, but not easy to convert as a dictionary :("

If you had a (nice) way to populate such a context, would it be ok for you?

[benoitbryon]

A second workaround could be related to the execution context running the healthchecks.
I mean, in your use case, the "host" information is shared by all the healthchecks run on a machine.
At the moment, I have no nice idea about how we could do it. Perhaps using logging, but I'm not sure it would be easy to have relationships between an AssertionError and a log that was emitted before.
What about working on #22 first, then see if it could be used or adapted to your use case?

[benoitbryon]

you can interconnect different hosts running hospital tests for example.

This is a story on its own!
At the moment, I would tend to do something like that:

• have several machines, each one running a healthcheck service
• have a "supervisor" machine, which collects and aggregates healthchecks on other machines

With such an architecture, on the "supervisor" (how would you name it?) you could easily have a result like that:

{
    "status": "pass",
    "details": [
        {
            "test": "GET http://api.nu.nl/healthchecks returns HTTP 200 OK",
            "status": "pass"
        },
        {
             "test": "GET http://192.168.1.44:1515 returns HTTP 200 OK",
            "status": "pass"
        },
        {
            "test": "GET http://192.168.1.55 returns HTTP 200 OK",
            "status": "pass"
        }
    ],
    "summary": {
        "skip": 0,
        "pass": 3,
        "expected_failure": 0,
        "error": 0,
        "fail": 0,
        "total": 3,
        "unexpected_success": 0
    }
}

If we had some hospital.assert_external_hospital(url) method, we could do it easily and have a better message in case of error, because it could take advantage of the JSON output. And it could highlight the information about external host (IP, port...).

Would this fit your use case?

That said, I also think the behaviour of such a "supervisor" should be:

• run "smoketests" on each node, i.e. fail fast.
• in case of a fail, optionally start a diagnosis, in a second process, i.e. run complete healthcheck on the host which failed, but as another action.

I think we should have at least two levels of feedback:

• smoketests => runs fast, basically gives fail/pass feedback
• diagnosis => runs slower, provides detailed feedback about the errors

With this idea in mind, I am not sure, at the moment, that a "supervisor" has to be able to concatenate results of every nodes, i.e. display details of each node. Whereas I think it has to be able to summarize results of each node, i.e. display fail/pass status of each node. If you need details about one node, isn't asking the node enough?

[benoitbryon]

And also, notice that we already can attach a custom message to assertions, but it is not easy to parse in the JSON output (it is a string, not a dictionary).
assert expression, "your custom message".
Perhaps it is enough, at least until we have better options.

[benoitbryon]

@bartee: the comments/questions above are meant to have a better idea about your needs, so that we can focus on what really matters ;)
Tell me if I am misunderstanding or forgetting some point!

[bartee]

Wooah, you've been busy! 👍

Interesting thoughts! After reading all this, I've gotten my hands dirty myself, and solved it for my usecase. First, I modified my testcase a bit, providing details:

@hospital.healthcheck
class RedisHealthCheck(unittest.TestCase):
    """
    Test the current state of Redis
    """
    def test_ping(self):
        r = redis.Redis(host=redis_host, port=redis_port,password=redis_password)
        try:
            r.ping()
        except redis.ConnectionError, e:
            self.fail(e)

    @property
    def details(self):
        return {
            'host': redis_host,
            'port': redis_port,
            'password': redis_password
        }

After that, I've added two lines to the details-method of the HealthCheckApp- class:

class HealthCheckApp(HealthCheckProgram):

    def details(self, result):
        """Generate report details dict."""
        for status, test, context in result.results:
            report = {
                'test': self.get_test_title(test),
                'status': status,
            }
            if context is not None:
                report['context'] = u'{context}'.format(context=context)

            if hasattr(test,'details'):
                report['details'] = test.details

            yield report

That works like a charm. In that way, the supervisor can use those details.details of a test case to interconnect different hosts. And you don't have to change the signature of existing output (or parse a string into something you can work with :-))

Only downside I can see: I can imagine details.details being a bit of a confusing name. But on the other hand, there might be use cases out there where other kinds of test specs are necessary.

I've already built a little supervisor-project myself. It's config based, checks a configured set of hosts for hospital-based output and puts that output into Redis. I yet have to plug it into my hosts-dashboard. Somewhere soon I'll put that little thing on my github.

[bartee]

About that supervisor: https://github.com/bartee/hospital_supervisor