Test failure: `test_permission_for_files_anonymous_func_migration_api`

[github-actions]

❌ test_permission_for_files_anonymous_func_migration_api: TimeoutError: Timed out after 0:02:00 (12m15.739s)

TimeoutError: Timed out after 0:02:00
[gw0] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) starting
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) progress=2(+2)
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) finished
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account)
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:15 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-4e814949fcfb3ac6cee5f521596c964f-b329ab152a4a2383-00]
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) starting
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) progress=2(+2)
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account) finished
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-IuPLq24N-ra78c29d0c (workspace) -> sdk-sKiPAGDE-ra78c29d0c (account)
06:03 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:15 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-4e814949fcfb3ac6cee5f521596c964f-b329ab152a4a2383-00]
[gw0] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #771}

[github-actions]

❌ test_permission_for_files_anonymous_func_migration_api: TimeoutError: Timed out after 0:02:00 (12m22.597s)

TimeoutError: Timed out after 0:02:00
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-PabEX780-ra78c29e38: https://DATABRICKS_HOST#setting/accounts/groups/2122739556119625
06:01 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-vVl8hM8l-ra78c29e38: https://accounts.CLOUD_ENVdatabricks.net/users/groups/150944127735316/members
[gw2] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-PabEX780-ra78c29e38: https://DATABRICKS_HOST#setting/accounts/groups/2122739556119625
06:01 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-vVl8hM8l-ra78c29e38: https://accounts.CLOUD_ENVdatabricks.net/users/groups/150944127735316/members
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) starting
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) progress=2(+2)
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) finished
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account)
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:01 INFO [databricks.labs.pytester.fixtures.baseline] Created hive_metastore.dummy_sfw9annwt schema: https://DATABRICKS_HOST/#explore/data/hive_metastore/dummy_sfw9annwt
06:01 INFO [databricks.labs.pytester.fixtures.baseline] Created ucx_sHlgHYdcsiXyQmU3 catalog: https://DATABRICKS_HOST/#explore/data/ucx_sHlgHYdcsiXyQmU3
06:13 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-2624c141c63e49e31aea486a2cc1a880-8e929a2cc53f4387-00]
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:01 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-PabEX780-ra78c29e38: https://DATABRICKS_HOST#setting/accounts/groups/2122739556119625
06:01 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-vVl8hM8l-ra78c29e38: https://accounts.CLOUD_ENVdatabricks.net/users/groups/150944127735316/members
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) starting
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) progress=2(+2)
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account) finished
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-PabEX780-ra78c29e38 (workspace) -> sdk-vVl8hM8l-ra78c29e38 (account)
06:01 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:01 INFO [databricks.labs.pytester.fixtures.baseline] Created hive_metastore.dummy_sfw9annwt schema: https://DATABRICKS_HOST/#explore/data/hive_metastore/dummy_sfw9annwt
06:01 INFO [databricks.labs.pytester.fixtures.baseline] Created ucx_sHlgHYdcsiXyQmU3 catalog: https://DATABRICKS_HOST/#explore/data/ucx_sHlgHYdcsiXyQmU3
06:13 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-2624c141c63e49e31aea486a2cc1a880-8e929a2cc53f4387-00]
[gw2] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #774}

[github-actions]

❌ test_permission_for_files_anonymous_func_migration_api: TimeoutError: Timed out after 0:02:00 (16m29.868s)

TimeoutError: Timed out after 0:02:00
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-ctt1I8bn-ra78c29e9c: https://DATABRICKS_HOST#setting/accounts/groups/2126332687182356
06:11 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-DLD0kvvb-ra78c29e9c: https://accounts.CLOUD_ENVdatabricks.net/users/groups/155766821258354/members
[gw4] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-ctt1I8bn-ra78c29e9c: https://DATABRICKS_HOST#setting/accounts/groups/2126332687182356
06:11 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-DLD0kvvb-ra78c29e9c: https://accounts.CLOUD_ENVdatabricks.net/users/groups/155766821258354/members
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) starting
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) progress=2(+2)
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) finished
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account)
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:11 INFO [databricks.labs.pytester.fixtures.baseline] Created hive_metastore.dummy_s9q8far08 schema: https://DATABRICKS_HOST/#explore/data/hive_metastore/dummy_s9q8far08
06:11 INFO [databricks.labs.pytester.fixtures.baseline] Created ucx_8dDkDZQjW00E1C3P catalog: https://DATABRICKS_HOST/#explore/data/ucx_8dDkDZQjW00E1C3P
06:27 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-8562f3ff58159e380f0b2d2035e935a8-744e0fd1c13b34bd-00]
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.sdk] Using Databricks Metadata Service authentication
06:11 INFO [databricks.labs.pytester.fixtures.iam] Workspace group sdk-ctt1I8bn-ra78c29e9c: https://DATABRICKS_HOST#setting/accounts/groups/2126332687182356
06:11 INFO [databricks.labs.pytester.fixtures.iam] Account group sdk-DLD0kvvb-ra78c29e9c: https://accounts.CLOUD_ENVdatabricks.net/users/groups/155766821258354/members
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions for 1 account groups.
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) starting
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) progress=2(+2)
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrating permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account) finished
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions: sdk-ctt1I8bn-ra78c29e9c (workspace) -> sdk-DLD0kvvb-ra78c29e9c (account)
06:11 INFO [databricks.labs.ucx.workspace_access.groups] Migrated 2 permissions for 1/1 groups successfully.
06:11 INFO [databricks.labs.pytester.fixtures.baseline] Created hive_metastore.dummy_s9q8far08 schema: https://DATABRICKS_HOST/#explore/data/hive_metastore/dummy_s9q8far08
06:11 INFO [databricks.labs.pytester.fixtures.baseline] Created ucx_8dDkDZQjW00E1C3P catalog: https://DATABRICKS_HOST/#explore/data/ucx_8dDkDZQjW00E1C3P
06:27 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-8562f3ff58159e380f0b2d2035e935a8-744e0fd1c13b34bd-00]
[gw4] linux -- Python 3.10.19 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from nightly #775}