Automated tests not stable - or... is the app not stable when interacting quickly?

[mvanbrab]

Say 20% of the time, when running the automated tests (in the loama subproject), a random number of tests fail.

In all cases, the cause is the same: the initial contents do not appear:

Error: Timed out 5000ms waiting for expect(locator).toBeVisible()

Locator: getByRole('link', { name: 'Resources', exact: true })
Expected: visible
Received: <element(s) not found>
Call log:
  - expect.toBeVisible with timeout 5000ms
  - waiting for getByRole('link', { name: 'Resources', exact: true })


   at auth.setup.ts:21

  19 |
  20 |
> 21 |     await expect(page.getByRole('link', { name: 'Resources', exact: true })).toBeVisible();
     |                                                                              ^
  22 |     await expect(page.getByRole('main')).toContainText('home');
  23 | }
  24 |
    at authSetup (/home/mavbraba/projects/solid/loama/loama/tests/auth.setup.ts:21:78)

And no, increasing the timeout does not solve this :-)

[mvanbrab]

TL;DR

Below an overview of what I tested, trying to solve this.

In the end, I'm not sure whether the cause is in the playwright testing framework or in the app.

For now I have disabled automated testing in github ci by renaming loama/.github/workflows/playwright.yml to loama/.github/workflows/playwright.yml.disabled and by modifying the README files.

Tested

Isolate on an otherwise idle linux machine

(I was suspecting my local machine)

Not the solution.

Increase timeout of the specific expect call:

Modified the crucial line 22 in loama/test/auth.setup.ts to:

await expect(page.getByRole('link', { name: 'Resources', exact: true })).toBeVisible({timeout: 30_000});

Not the solution.

Increase global expect timeout:

Add this to loama/playwright.config.ts:

expect: {
    timeout: 30_000
},

Not the solution.

Wait for the IDP to redirect back

Add a line in front of line 22 in loama/test/auth.setup.ts. New lines:

await page.waitForURL(/http:\/\/localhost:4173/);
await expect(page.getByRole('link', { name: 'Resources', exact: true })).toBeVisible({timeout: 30_000});

Not the solution.

Version bumps

In package.json:

• "@inrupt/solid-client": from "^2.1.0" to "^2.1.2",
• "@inrupt/solid-client-authn-browser": from "^2.2.4" to "^2.3.0",

In loama/package.json:

• "@solid/community-server": from "^7.1.2" to "^7.1.6"

Not the solution;

Inline authSetup

Replace test.beforeEach(authSetup); by copying the code as first lines of each separate test in loama/tests/permission-table.spec.ts and loama/tests/resource-explorer.spec.ts.

Very strange: this improved the failure rate from 1 out of 3 to 1 out of 10, finally resulting in the same error one more time.

So the verdict remains: not the solution.