Could you explain on why `t.Cleanup` is required over `defer` in some situations?

[brandur]

tparallel will require the use of t.Cleanup instead of defer in cases where a top-level test case has subtests, and that top-level test case is t.Parallel. From the README:

Although t.Parallel() is called in the sub-test function, it is post-processed by defer instead of t.Cleanup()

I was bringing t.Parallel into a project and had to make some changes related to this, but was having a hard time explaining to my colleague why exactly those changes are necessary.

Could you elaborate a little more on why exactly t.Cleanup needs to be used instead of defer in some cases? I can help expand the README to explain it if you'd like (after I understand it myself that is).

[telemachus]

I was curious too, so this is for future Googlers. The README says that this tool was inspired by a blog post: Go 言語でのテストの並列化 〜t.Parallel()メソッドを理解する〜. I used Google Translate to get the following translation of the last section, which is helpfully labeled "defer statement and t.Cleanup() method." (Two notes: (1) the chunk I translated refers to code from earlier in the blog post, but I didn't want to translate and copy everything. I think it's clear enough, but YMMV. (2) I tweaked the translation in a few cases where I thought it was gibberish. Refer back to the original article (and retranslate) as needed.)

When doing post-processing at the end of the test, you need to be careful whether you use a defer statement or a t.Cleanup() method. As for the top-level test functions, the basics are:

• If the top-level test function does not contain a subtest function with a t.Run() method, then either a defer statement or a t.Cleanup() method may be used to perform the post-processing.
• A defer statement or a t.Cleanup() method (both work) if a top-level test function contains subtest functions with a t.Run() method and none of its subtest functions call a t.Parallel() method. You can perform the post-processing either way.
• If a top-level test function contains subtest functions with t.Run() methods and at least one of those subtest functions calls the t.Parallel() method, then perform post-processing with the t.Cleanup() method.

A defer statement is called when the function containing the statement returns. Review the example execution of the preceding code. The Test_Func1 function returns before the Func1_Sub1 and Func1_Sub2 subtest functions have finished (Action 2). Therefore, the function with the delay specified in the defer statement included in the Test_Func1 function is called before the subtest functions of Func1_Sub1 and Func1_Sub2 resume processing after being paused (in the above execution result, note the position of Test_Func1 returned).

For example, even if the top-level test function delays a post-processing call, such as deleting a record in a table created by the subtest function, with a defer statement, if the subtest function calls the t.Parallel() method, the post-processing function specified in the defer statement will be called prior to the execution of the subtest function. In such cases, write the cleanup using the t.Cleanup() method instead of the defer statement.

The specification of the t.Cleanup() method is as follows.

func (c *T) Cleanup(f func())
Cleanup registers a function to be called when the test and all its subtests complete. Cleanup functions will be called in last added, first called order.

The function registered with the t.Cleanup() method is said to be called when all subtests are finished.

Then, what happens to the post-processing in the subtest function described by the t.Run() method is as follows (basically the same as the above three items, but the subject is different).

• If the subtest function does not contain further subsubtest functions with nested t.Run() methods, then either the defer statement or the t.Cleanup() method may be used to describe the post-processing.
• A defer statement or t.Cleanup() if a subtest function contains further subsubtest functions with nested t.Run() methods and none of the subsubtest functions call the t.Parallel() method. You can write post-processing in either method.
• If the sub-test function contains further sub-sub-test functions with nested t.Run() methods and at least one sub-sub-test function calls the t.Parallel() method, then perform post-processing with the t.Cleanup() method.

If you are too lazy to memorize the above six items about top-level test functions and sub-test functions, some projects say, "In test code that uses the t.Parallel() method, the post-processing is t.Cleanup( )."

[brandur]

Ah, thanks for tracking that down @telemachus. I'm going to add a couple snippets from the blog post here as well for easier reference.

The implementation of Func1_Sub1 and Func1_Sub2 referenced above:

func Test_Func1(t *testing.T) {
    defer trace("Test_Func1")()

    t.Run("Func1_Sub1", func(t *testing.T) {
        defer trace("Func1_Sub1")()
        t.Parallel()

        // ...
    })

    t.Run("Func1_Sub2", func(t *testing.T) {
        defer trace("Func1_Sub2")()

        t.Parallel()
        // ...
    })

    // ...
}

And a sample run where you can see the top-level Test_Func1 defer running before the subtests finish:

=== RUN   Test_Func1
Test_Func1 entered
=== RUN   Test_Func1/Func1_Sub1
Func1_Sub1 entered                          <- Func1_Sub1が開始
=== PAUSE Test_Func1/Func1_Sub1             <- Func1_Sub1が一時停止
=== RUN   Test_Func1/Func1_Sub2
Func1_Sub2 entered                          <- Func1_Sub2が開始
=== PAUSE Test_Func1/Func1_Sub2             <- Func1_Sub2が一時停止
Test_Func1 returned                         <- Test_Func1の呼び出し戻り（＊）
=== CONT  Test_Func1/Func1_Sub1             <- Func1_Sub1が再開
Func1_Sub1 returned                         <- Func1_Sub1が完了
=== CONT  Test_Func1/Func1_Sub2             <- Func1_Sub2が再開
Func1_Sub2 returned                         <- Func1_Sub2が完了
--- PASS: Test_Func1 (0.00s)                <- Test_Func1の結果表示
    --- PASS: Test_Func1/Func1_Sub1 (0.00s)
    --- PASS: Test_Func1/Func1_Sub2 (0.00s)
=== RUN   Test_Func2                        <- ここまでTest_Func2は実行されない
Test_Func2 entered
=== PAUSE Test_Func2
=== RUN   Test_Func3
Test_Func3 entered
Test_Func3 returned
--- PASS: Test_Func3 (0.00s)
=== RUN   Test_Func4
Test_Func4 entered
=== PAUSE Test_Func4
=== RUN   Test_Func5
Test_Func5 entered
Test_Func5 returned
--- PASS: Test_Func5 (0.00s)
=== CONT  Test_Func2
Test_Func2 returned
=== CONT  Test_Func4
Test_Func4 returned
--- PASS: Test_Func4 (0.00s)
--- PASS: Test_Func2 (0.00s)
PASS

If I had to summarize the problem: use of t.Parallel() in tests may cause them to be paused while the runner is working on other things. While paused, a parent test function may return, which would execute any defer statements it had registered. Use of t.Cleanup is the workaround because although it behaves similarly to a defer, it's baked into the test framework so that it can guarantee the right thing will happen.