Bug resolve synpase profiler errors by eri-adepoju · Pull Request #2289 · databrickslabs/lakebridge

eri-adepoju · 2026-02-12T21:44:36Z

Changes

What does this PR do?

Pipeline and trigger run extraction – Updates handling of list_pipeline_runs and list_trigger_runs so they correctly process batched yields (lists of dicts) instead of treating each yield as a single run.
Serverless SQL pool routines – Adds list_serverless_routines() using sys.objects because information_schema.routines is not available in serverless pools.
Server-level DMVs – Reconnects to the master database before querying server-level DMVs (e.g., data_processed), since these views must be queried from master.
Whitespace in credentials – Strips leading/trailing whitespace from credentials and config values (user, password, server, database, driver, auth_type, tz_info, development_endpoint) to avoid connection failures from copy-paste or config issues.
DataFrame concatenation – Replaces deprecated DataFrame.union() with pd.concat() in monitoring_metrics_extract.py.
Documentation – Clarifies Azure auth (DefaultAzureCredential order), local vs CI setup, DMV permissions (VIEW DATABASE STATE, VIEW SERVER STATE, VIEW DEFINITION), and serverless pool catalog views.

Relevant implementation details

serverless_sqlpool_extract.py: Uses get_sqlpool_reader(config, 'master', ...) before querying server-level DMVs; routines query switched from list_routines to list_serverless_routines.
database_manager.py: All credential/config string fields use .strip() before connection
monitoring_metrics_extract.py: step_name for spark pool metrics moved outside the loop, so it is defined even when the loop is empty.

Linked issues

Resolves #2287

Functionality

added relevant user documentation
added new CLI command
modified existing command: databricks labs lakebridge ...
fixed existing functionality

Tests

manually tested
added unit tests
added integration tests

…from pip command to enable debugging.

…error_handler

…ice principal roles.

…on_schema.routines is not available

…`list_trigger_runs` rather than expecting one dict per run.

…for Dynamic Management Views.

…ames. Add unit tests to verify that whitespace in credential fields and batch processing of pipeline and trigger runs are is correctly handled. Replace union with concat for Pandas Dataframes.

…o bug_resolve_synpase_profiler_errors

…ists correctly

…abase, and driver information.

…rofiler_errors

… a dictionary.

codecov · 2026-02-12T21:58:52Z

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.41%. Comparing base (6bd912f) to head (ee74132).

Files with missing lines	Patch %	Lines
...ge/resources/assessments/synapse/common/queries.py	0.00%	3 Missing ⚠️
.../assessments/synapse/serverless_sqlpool_extract.py	0.00%	2 Missing ⚠️
...resources/assessments/synapse/workspace_extract.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2289      +/-   ##
==========================================
- Coverage   66.44%   66.41%   -0.03%     
==========================================
  Files          99       99              
  Lines        9089     9093       +4     
  Branches      974      974              
==========================================
  Hits         6039     6039              
- Misses       2874     2878       +4     
  Partials      176      176

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-02-12T22:00:20Z

✅ 143/143 passed, 5 flaky, 5 skipped, 34m16s total

Flaky tests:

🤪 test_installs_and_runs_pypi_bladebridge (26.645s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (16.774s)
🤪 test_transpiles_informatica_to_sparksql_non_interactive[True] (16.943s)
🤪 test_transpile_teradata_sql (6.998s)
🤪 test_transpile_teradata_sql_non_interactive[False] (5.811s)

_{Running from acceptance #3944}

sundarshankar89

Thank you @eri-adepoju for identifying gaps in the extraction, and for the authentication related changes I will use different approach, I can PR that sperately.

src/databricks/labs/lakebridge/connections/database_manager.py

sundarshankar89 · 2026-02-13T02:15:46Z

docs/lakebridge/docs/assessment/profiler/synapse.mdx

 ```
+
+The profiler uses Azure SDK's `DefaultAzureCredential` which attempts authentication in this order:
+1. **Environment Variables** (Service Principal):


Good catch, Thanks for this document update.

docs/lakebridge/docs/assessment/profiler/synapse.mdx

src/databricks/labs/lakebridge/resources/assessments/synapse/workspace_extract.py

sundarshankar89 · 2026-02-13T02:26:07Z

tests/unit/assessment/test_extraction_whitespace.py

+def test_zoneinfo_creation_with_stripped_whitespace() -> None:
+    """Test that zoneinfo.ZoneInfo works correctly with stripped timezone strings."""
+    # This tests the core behavior that our code relies on
+    tz_with_whitespace = ' America/New_York '


Now I see what has happened, I will tackle this differently for now you cna remove .strip() I will ensure the .credentials.yml doesn't have any spaces likes these when stored.

I dont think these tests are vaild since we removed strip

sundarshankar89 · 2026-02-13T02:28:28Z

tests/unit/assessment/test_workspace_extract_batch_handling.py

+from datetime import date
+
+
+def test_pipeline_runs_handles_batches_correctly():


Thanks for adding tests, I m adding type hints in this PR, having type hints enabled and then having tests will help our case.

#2264

tests/unit/assessment/test_workspace_extract_batch_handling.py

tests/unit/assessment/test_workspace_extract_json_normalize.py

src/databricks/labs/lakebridge/connections/database_manager.py

goodwillpunning · 2026-02-13T15:02:36Z

docs/lakebridge/docs/assessment/profiler/synapse.mdx


 - Profiler uses the Python version of Azure SDK libraries to extract information about target Synapse Workspace.
- For making the Azure API calls using Azure SDK you need an Azure Service Principal with the following role assignments.
+- For making the Azure API calls using Azure SDK, the authenticated identity (user or service principal) needs the following role assignments.


Same as above - let's remove mention of service principal until support is added in a future PR.

This is separate from a service principal accessing information schema tables. Service principals can access Synapse workspaces and Azure monitor metrics with the profiler today.

src/databricks/labs/lakebridge/connections/database_manager.py

goodwillpunning · 2026-02-13T15:05:51Z

src/databricks/labs/lakebridge/resources/assessments/synapse/common/queries.py

                   """

+    @staticmethod
+    def list_serverless_routines(pool_name, redact_sql_text: bool = False) -> str:


…fensive logs and module comments.

sundarshankar89 · 2026-02-18T15:39:26Z

@eri-adepoju can you fix fmt errors.

sundarshankar89 · 2026-02-20T09:19:19Z

@eri-adepoju there is small conflict can resolve those and make it ready for review changes look good to me.

eri-adepoju · 2026-02-24T18:46:57Z

@eri-adepoju there is small conflict can resolve those and make it ready for review changes look good to me.

All done!

gueniai

LGTM

goodwillpunning

LGTM!

Sundar approved changes in a previous comment.

eri-adepoju and others added 18 commits January 13, 2026 14:30

Update the dependency installation process to capture and log output …

7ad999d

…from pip command to enable debugging.

Merge remote-tracking branch 'origin/main' into bug_improve_profiler_…

1bac6ad

…error_handler

Clarify local development and CI environment setups for user and serv…

d6485af

…ice principal roles.

Add method to list serverless routines in SynapseQueries as informati…

6d01f89

…on_schema.routines is not available

Update logic to process lists of dicts from list_pipeline_runs and …

c82cd88

…`list_trigger_runs` rather than expecting one dict per run.

Reconnect to master for server-level Dynamic Management Views.

4274db6

Add Azure authentication validation and clarify required permissions …

06437da

…for Dynamic Management Views.

Strip leading and trailing whitespace from credentials and database n…

c6c0877

…ames. Add unit tests to verify that whitespace in credential fields and batch processing of pipeline and trigger runs are is correctly handled. Replace union with concat for Pandas Dataframes.

Remove Azure auth verification

3a7fded

initial commit

e15098c

fixed path

85da6a6

Merge branch 'main' into patch/profiler_extract_path

47f1f31

Merge remote-tracking branch 'origin/patch/profiler_extract_path' int…

8ca7b95

…o bug_resolve_synpase_profiler_errors

Update son_normalize usage in workspace_extract.py to handle nested l…

b6bc49c

…ists correctly

Added detailed logging for connection attempts, including server, dat…

12f1f2d

…abase, and driver information.

Merge remote-tracking branch 'origin/main' into bug_resolve_synpase_p…

3377259

…rofiler_errors

Revert temporary logging

d91a080

Added validation to ensure each entry in pipeline and trigger runs is…

8293ea4

… a dictionary.

eri-adepoju requested a review from a team as a code owner February 12, 2026 21:44

Merge branch 'main' into bug_resolve_synpase_profiler_errors

d078d64

eri-adepoju had a problem deploying to tool February 12, 2026 21:50 — with GitHub Actions Error

Revert logging change as it's handled in a separate PR

214b223

eri-adepoju temporarily deployed to tool February 12, 2026 21:54 — with GitHub Actions Inactive

Resolve formatting errors

bfdec3b

eri-adepoju temporarily deployed to tool February 12, 2026 22:00 — with GitHub Actions Inactive

Merge branch 'main' into bug_resolve_synpase_profiler_errors

08b1542

eri-adepoju had a problem deploying to tool February 12, 2026 22:20 — with GitHub Actions Error

Remove unused variable

42771c9

eri-adepoju temporarily deployed to tool February 12, 2026 22:26 — with GitHub Actions Inactive

sundarshankar89 previously requested changes Feb 13, 2026

View reviewed changes

goodwillpunning requested changes Feb 13, 2026

View reviewed changes

eri-adepoju had a problem deploying to tool February 16, 2026 20:13 — with GitHub Actions Error

Reverting strip addition for more comprehensive fix and removal of de…

c569709

…fensive logs and module comments.

eri-adepoju force-pushed the bug_resolve_synpase_profiler_errors branch from 12f5776 to c569709 Compare February 16, 2026 20:23

eri-adepoju had a problem deploying to tool February 16, 2026 20:23 — with GitHub Actions Error

eri-adepoju requested review from goodwillpunning and sundarshankar89 February 16, 2026 20:26

Merge branch 'main' into bug_resolve_synpase_profiler_errors

0f5317a

eri-adepoju temporarily deployed to tool February 16, 2026 20:26 — with GitHub Actions Inactive

Fix fmt isues and clarify language for az login

0379246

eri-adepoju temporarily deployed to tool February 18, 2026 22:45 — with GitHub Actions Inactive

Merge branch 'main' into bug_resolve_synpase_profiler_errors

cede5e9

eri-adepoju temporarily deployed to tool February 24, 2026 17:00 — with GitHub Actions Inactive

eri-adepoju added 2 commits February 24, 2026 11:05

Delete obsolete test.

2f7e921

Replace obsolete function call.

ee74132

eri-adepoju temporarily deployed to tool February 24, 2026 17:41 — with GitHub Actions Inactive

gueniai approved these changes Feb 24, 2026

View reviewed changes

goodwillpunning approved these changes Feb 25, 2026

View reviewed changes

Merge branch 'main' into bug_resolve_synpase_profiler_errors

1152523

sundarshankar89 deployed to tool February 25, 2026 12:26 — with GitHub Actions Active

gueniai enabled auto-merge February 26, 2026 00:53

gueniai added this pull request to the merge queue Feb 26, 2026

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Feb 26, 2026

		from datetime import date


		def test_pipeline_runs_handles_batches_correctly():

Conversation

eri-adepoju commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

What does this PR do?

Relevant implementation details

Linked issues

Functionality

Tests

Uh oh!

codecov bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Feb 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sundarshankar89 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sundarshankar89 commented Feb 18, 2026

Uh oh!

sundarshankar89 commented Feb 20, 2026

Uh oh!

eri-adepoju commented Feb 24, 2026

Uh oh!

gueniai left a comment

Choose a reason for hiding this comment

Uh oh!

goodwillpunning left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eri-adepoju commented Feb 12, 2026 •

edited

Loading

codecov bot commented Feb 12, 2026 •

edited

Loading

github-actions bot commented Feb 12, 2026 •

edited

Loading