Skip to content

Commit 6692f7d

Browse files
ObadaSObada Haddad
ObadaS
and
Obada Haddad
authored
Initial Playwright integration (#2042)
* Initial Playwright integration * Circle-ci fix * Further Circle-ci fix * Further Circle-ci fix * circle-ci test removal * circle-ci tests * circle-ci tests * circle-ci tests * circle-ci tests * github workflow for tests on selfhosted runner * github workflow fixes * github workflow fixes * github workflow fixes * github workflow fixes * workflow changes * some workflow tests * circle-ci tests * Trying Yarn instead of NPM and NPX * Trying Yarn instead of NPM and NPX for circle-ci * Trying Yarn instead of NPM and NPX for github * Fixed circle-ci error * Trying again on Self hosted runner * Reorganized worlflow file * Reorganized worlflow files * Fixed circleci job name * circleci ressource class * circleci ressource class * change playwright to use pytest instead of npx * fix uv location * fix compose file location * fix compose exec command; change circle-ci config * disable old playwright tests using yarn * fix circle-ci test; update uv.lock * fix psycopg dependencies; possible fix for playwright browser dependencies * change to github hosted runner * add competition tests; update account creation, submission tests * try localhost:8000 instead of localhost as baseurl * circle-ci changes * fix docker pull * circle changes * circle changes * circle-ci .env changes * removed docker compose selenium since we don't need it anymore * update circle-ci image version from ubuntu 22 to 24 * update tests * change test order * add page reload * change waiting for finished logic * update test * remove failing tests for now * cleanup * more cleanup * even more cleanup * even more cleanup v2 * change order of circleci test to potentially save time * add leaderboard test for submissions; make the pytest skip more intelligent * fix poetry lock accidental update * fix test failing by adding exact=True * simplify submissions tests; ruff formatting * update documentation, remove selenium docker-compose file * add --frozen to uv lock; make packages in the pyproject.toml use ~ instead of > * update documentation and README.md --------- Co-authored-by: Obada Haddad <obada.haddad@lisn.fr>
1 parent 3036a5a commit 6692f7d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+1445
-589
lines changed

‎.circleci/config.yml‎

Lines changed: 47 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,10 @@ version: 2
33
jobs:
44
test:
55
machine:
6-
image: ubuntu-2204:2024.01.2
6+
image: ubuntu-2404:2025.09.1
7+
resource_class: large
78
steps:
89
- checkout
9-
1010
- run:
1111
# NOTE: To connect to this, use an SSH tunnel in front, like so..
1212
#
@@ -22,54 +22,73 @@ jobs:
2222
# x11vnc -forever -nopw
2323
background: true
2424

25-
- run:
25+
- run:
2626
name: "Setup: Copy environment variables"
2727
command: cp .env_circleci .env
2828

29-
3029
- run:
3130
name: "Setup: Create directories for MinIO (cannot be made by docker for some reason)"
3231
command: |
33-
sudo mkdir -p var/minio/public
34-
sudo mkdir -p var/minio/private
32+
mkdir -p var/minio/public
33+
mkdir -p var/minio/private
3534
3635
- run:
37-
name: "Docker: Build containers and collect static files"
36+
name: "Setup: Prepare the playwright environment"
3837
command: |
39-
docker compose -f docker-compose.yml -f docker-compose.selenium.yml up -d
40-
docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django python manage.py collectstatic --noinput
41-
42-
- run:
43-
name: "Lint: Check code style with flake8"
44-
command: docker-compose exec django flake8 src/
45-
46-
47-
- run:
38+
cd tests
39+
apt update && apt upgrade -y
40+
curl -LsSf https://astral.sh/uv/install.sh | sh
41+
$HOME/.local/bin/uv sync --frozen
42+
$HOME/.local/bin/uv run playwright install
43+
- run:
4844
name: "Docker: Pull required images"
4945
# not available without "not e2e" tests as they pull ahead of time
5046
command: |
51-
docker pull codalab/codalab-legacy:py37
52-
docker pull codalab/codalab-legacy:py3
53-
docker pull vergilgxw/autotable:v2
47+
docker pull codalab/codalab-legacy:py37
48+
docker pull codalab/codalab-legacy:py3
49+
docker pull vergilgxw/autotable:v2
50+
background: true
51+
52+
- run:
53+
name: "Docker: Build containers and collect static files"
54+
command: |
55+
docker compose up -d
56+
docker compose exec django python manage.py collectstatic --noinput
57+
docker compose exec django python manage.py migrate
58+
docker compose exec django python ./manage.py createsuperuser --no-input
59+
60+
- run:
61+
name: "Get compute worker, site worker and django logs"
62+
command: |
63+
mkdir dockerLogs
64+
docker compose logs -f site_worker compute_worker django > dockerLogs/django_workers.log
65+
background: true
66+
67+
- run:
68+
name: "Lint: Check code style with flake8"
69+
command: docker compose exec django flake8 src/
5470

5571
- run:
5672
name: "Tests: Run unit/integration tests (excluding e2e)"
57-
command: docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django py.test src/ -m "not e2e"
73+
command: docker compose exec django py.test src/ -m "not e2e"
5874

75+
# We give the name of the test files manually because we need test_auth.py to be run before the others for state.json file to be created
76+
# CI="true" to skip some tests that fail in the CI for now
5977
- run:
6078
name: "Tests: Run end-to-end (E2E) tests"
61-
command: docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django py.test src/tests/functional/ -m e2e
62-
no_output_timeout: 60m
79+
command: |
80+
cd tests && CI="true" $HOME/.local/bin/uv run pytest test_auth.py test_account_creation.py test_competition.py test_submission.py
81+
no_output_timeout: 30m
6382

6483
# Example to run specific set of tests (for debugging individual tests from a batch of tests)
6584
# - run:
66-
# name: e2e tests - competitions
67-
# command: docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django py.test src/tests/functional/test_competitions.py -m e2e
68-
# no_output_timeout: 60m
69-
70-
85+
# name: "Tests: Run end-to-end (E2E) tests"
86+
# command: cd tests && $HOME/.local/bin/uv run pytest test_auth.py test_competition.py
87+
# no_output_timeout: 30m
88+
- store_artifacts:
89+
path: tests/test-results
7190
- store_artifacts:
72-
path: artifacts/
91+
path: dockerLogs/
7392

7493
workflows:
7594
version: 2

‎.env_circleci‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,9 @@ AWS_STORAGE_PRIVATE_BUCKET_NAME=private
3232
# NOTE! port 9000 here should match $MINIO_PORT
3333
AWS_S3_ENDPOINT_URL=http://172.17.0.1:9000/
3434
AWS_QUERYSTRING_AUTH=False
35+
DJANGO_SUPERUSER_PASSWORD=codabench
36+
DJANGO_SUPERUSER_EMAIL=test@test.com
37+
DJANGO_SUPERUSER_USERNAME=codabench
38+
DOMAIN_NAME=localhost:80
39+
TLS_EMAIL=your@email.com
40+
SUBMISSIONS_API_URL=http://django:8000/api

‎.github/workflows/tests.yml.DISABLED‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
name: build_and_test
2+
on: [push]
3+
jobs:
4+
build:
5+
name: Build necessary services
6+
# runs-on: self-hosted
7+
runs-on: ubuntu-latest
8+
steps:
9+
- name: Check out repository code
10+
uses: actions/checkout@v5
11+
- name: "Setup: Copy environment variables"
12+
run: cp .env_circleci .env
13+
- name: "Setup: Create directories for MinIO (cannot be made by docker for some reason)"
14+
run: |
15+
mkdir -p var/minio/public
16+
mkdir -p var/minio/private
17+
- name: "Setup: Prepare the playwright environment"
18+
run: |
19+
cd playwrightPython
20+
curl -LsSf https://astral.sh/uv/install.sh | sh
21+
$HOME/.local/bin/uv sync
22+
$HOME/.local/bin/uv run playwright install
23+
- name: "Docker: Build containers and collect static files"
24+
run: |
25+
docker compose -f docker-compose.yml -f docker-compose.selenium.yml up -d
26+
docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django python manage.py collectstatic --noinput
27+
docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django python manage.py migrate
28+
docker compose -f docker-compose.yml exec django python ./manage.py createsuperuser --no-input
29+
- name: "Docker: Pull required images"
30+
run: |
31+
docker pull codalab/codalab-legacy:py37
32+
docker pull codalab/codalab-legacy:py3
33+
linter:
34+
name: Flake8 linter
35+
runs-on: self-hosted
36+
needs: [build]
37+
steps:
38+
- name: "Lint: Check code style with flake8"
39+
run: docker compose exec django flake8 src/
40+
unit_tests:
41+
name: Unit test with Selenium
42+
runs-on: self-hosted
43+
needs: [linter,build]
44+
steps:
45+
- name: "Tests: Run unit/integration tests (excluding e2e)"
46+
run: docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django py.test src/ -m "not e2e"
47+
e2e:
48+
name: End to End tests with Playwright
49+
runs-on: self-hosted
50+
needs: [linter,build]
51+
steps:
52+
- name: "Tests: Run end-to-end (E2E) tests"
53+
run: cd playwrightPython && $HOME/.local/bin/uv run pytest test_auth.py test_account_creation.py test_competition.py test_submission.py
54+
cleanup:
55+
name: Cleanup
56+
runs-on: self-hosted
57+
if: ${{ always() }}
58+
needs: [unit_tests,e2e,linter]
59+
steps:
60+
- name: Cleanup
61+
run: |
62+
docker compose -f docker-compose.yml -f docker-compose.selenium.yml down --rmi all
63+
rm -rf ${{ github.workspace }}/*

‎docker-compose.selenium.yml‎

Lines changed: 0 additions & 18 deletions
This file was deleted.

‎documentation/docs/Developers_and_Administrators/Adding-e2e-tests.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# To run the tests locally
2+
Install uv : [https://docs.astral.sh/uv/getting-started/installation/](https://docs.astral.sh/uv/getting-started/installation/)
3+
4+
Run the following commands:
5+
```bash
6+
cd tests
7+
uv sync --frozen
8+
uv run playwright install
9+
docker compose exec -e DJANGO_SUPERUSER_PASSWORD=codabench django python manage.py createsuperuser --username codabench --email codabench@test.mail --no-input
10+
uv run pytest test_auth.py test_account_creation.py test_competition.py test_submission.py
11+
```
12+
13+
# Adding Tests
14+
First, read the [documentation](https://playwright.dev/python/docs/writing-tests) on Playwright if you haven't used the tool before.
15+
Since we are using pytest, you should also try to get more familiar with it by reading some of its [documentation](https://docs.pytest.org/en/stable/getting-started.html).
16+
17+
18+
Once you are done, you can start adding tests. Playwright allows us to generate code with the following command :
19+
```bash
20+
uv run playwright codegen -o test.py
21+
```
22+
23+
This will open two windows:
24+
- A window containing the generated code
25+
- A browser that is used by playwright to generate the code. Every action you take there will generate new lines of the code.
26+
27+
Once you are done, close the browser and open the file that playwright created containing the code it generated. Make sure to test it to make it sure it works.
28+
29+
Since we are passing custom commands to pytest, we need to remove some of the generated lines. Spawning a new browser and/or context will make them not take into the commands we have added in the `pytest.ini` file :
30+
```python title="Original file created by codegen"
31+
from playwright.sync_api import Playwright, sync_playwright, expect
32+
33+
34+
def run(playwright: Playwright) -> None:
35+
browser = playwright.chromium.launch(headless=False)
36+
context = browser.new_context()
37+
page = context.new_page()
38+
page.goto("http://localhost/")
39+
page.get_by_text("Benchmarks/Competitions Datasets Login Sign-up").click()
40+
page.get_by_role("link", name="Login").click()
41+
42+
page.close()
43+
44+
# ---------------------
45+
context.close()
46+
browser.close()
47+
48+
49+
with sync_playwright() as playwright:
50+
run(playwright)
51+
```
52+
53+
The previous file becomes :
54+
```python title="Modified file"
55+
from playwright.sync_api import Page, expect
56+
57+
58+
def test_run(page: Page) -> None:
59+
page.goto("http://localhost/")
60+
page.get_by_text("Benchmarks/Competitions Datasets Login Sign-up").click()
61+
page.get_by_role("link", name="Login").click()
62+
```
63+

‎documentation/docs/Developers_and_Administrators/Running-tests.md‎

Lines changed: 6 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,22 +2,19 @@
22
# Without "end to end" tests
33
$ docker compose exec django py.test -m "not e2e"
44

5-
# "End to end tests" (a shell script to launch a selenium docker container)
6-
$ ./run_selenium_tests.sh
7-
8-
# If you are on Mac OSX it is easy to watch these tests, no need to install
9-
# anything just do:
10-
$ open vnc://0.0.0.0:5900
11-
12-
# And login with password "secret"
5+
# Playwright tests (make sure to install uv first: https://docs.astral.sh/uv/getting-started/installation/)
6+
uv sync --frozen
7+
uv run playwright install
8+
docker compose exec -e DJANGO_SUPERUSER_PASSWORD=codabench django python manage.py createsuperuser --username codabench --email codabench@test.mail --no-input
9+
uv run pytest test_auth.py test_account_creation.py test_competition.py test_submission.py
1310
```
1411

1512
## CircleCI
1613

1714
To simulate the tests run by CircleCI locally, run the following command:
1815

1916
```sh
20-
docker compose -f docker-compose.yml -f docker-compose.selenium.yml exec django py.test src/ -m "not e2e"
17+
docker compose -f docker-compose.yml exec django py.test src/ -m "not e2e"
2118
```
2219

2320
## Example competitions

‎documentation/mkdocs.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,7 @@ nav:
144144
- Backups - Automating Creation and Restoring: Developers_and_Administrators/Creating-and-Restoring-from-Backup.md
145145
- Submission Process Overview: Developers_and_Administrators/Submission-Process-Overview.md
146146
- Robot Submissions: Developers_and_Administrators/Robot-submissions.md
147+
- Adding Tests: Developers_and_Administrators/Adding-e2e-tests.md
147148
- Running Tests: Developers_and_Administrators/Running-tests.md
148149
- Automation: Developers_and_Administrators/Automating-with-Selenium.md
149150
- Manual Validation: Developers_and_Administrators/Manual-validation.md

‎poetry.lock‎

Lines changed: 1 addition & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎pyproject.toml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,6 @@ flake8 = "3.8.4"
6363
pytest = "6.2.1"
6464
pytest-django = "4.1.0"
6565
pytest-pythonpath = "0.7.3"
66-
selenium = "3.141.0"
6766
jinja2 = "3.1.4"
6867
requests = "2.32.2"
6968
drf-extra-fields = "3.0.2"

‎reset_db.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/usr/bin/env bash
22

3-
docker-compose exec db bash -c "
3+
docker compose exec db bash -c "
44
echo 'dropping database'
55
dropdb --if-exists -U \$DB_USERNAME \$DB_NAME &&
66
echo \$DB_PASSWORD &&
@@ -10,7 +10,7 @@ createdb -U \$DB_USERNAME \$DB_NAME &&
1010
echo 'create successful'
1111
exit" &&
1212

13-
docker-compose exec django bash -c "
13+
docker compose exec django bash -c "
1414
python manage.py migrate &&
1515
python manage.py generate_data
1616
exit"

0 commit comments

Comments
 (0)