@@ -140,10 +140,46 @@ cases have higher priority.
140140Finally, any hint caused by a test * error* rather than a test * failure* will receive
141141lower overall priority.
142142
143+ ### Tests should FAIL, not ERROR
144+
143145As much as possible, we want to write grading tests that result in a test
144146** failure** rather than an error. If your grading test errors given a student
145147solution, the test should be augmented with additional checks to result in a
146- failure instead.
148+ failure instead. Here is an example for a test that * could* error:
149+
150+ ```
151+ def test_x_is_positive(self):
152+ self.hint("x is not positive")
153+ self.assertGreater(implementation.x, 0)
154+ ```
155+
156+ In case the student wrote faulty code where ` implementation.x ` is, for example,
157+ a string, ` assertGreater ` will error, because '>' not supported between
158+ instances of 'str' and 'int'. So instead of failing, this test case would error.
159+ Thus, the test suite should check if ` x ` is a number before checking its value.
160+
161+ ```
162+ def test_x_is_positive(self):
163+ self.hint("x is not a number")
164+ self.assertIsInstance(implementation.x, numbers.Number)
165+ self.hint("x is not positive")
166+ self.assertGreater(implementation.x, 0)
167+ ```
168+
169+ or more broadly:
170+
171+ ```
172+ def test_x_is_positive(self):
173+ self.hint("x is not a positive number")
174+ try:
175+ self.assertGreater(implementation.x, 0)
176+ except:
177+ self.fail()
178+ ```
179+
180+ It often makes sense to implement such checks in a separate (non-test) method
181+ and to call that method in every test case. You may similarily be able to use
182+ ` setUp() ` , but if you do, make sure you call ` super().setUp() ` inside it.
147183
148184## Input function
149185
0 commit comments