Dev #208

Chenglong-MS · 2025-12-04T02:14:17Z

fix mysql connector issue
test pgsql connector issue
simplify UI

Bumps [vega](https://github.com/vega/vega) from 5.33.0 to 6.2.0. - [Release notes](https://github.com/vega/vega/releases) - [Commits](vega/vega@v5.33.0...v6.2.0) --- updated-dependencies: - dependency-name: vega dependency-version: 6.2.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

….2.0 Bump vega from 5.33.0 to 6.2.0

Add BigQuery DataLoader

Bumps [validator](https://github.com/validatorjs/validator.js) from 13.15.20 to 13.15.22. - [Release notes](https://github.com/validatorjs/validator.js/releases) - [Changelog](https://github.com/validatorjs/validator.js/blob/master/CHANGELOG.md) - [Commits](validatorjs/validator.js@13.15.20...13.15.22) --- updated-dependencies: - dependency-name: validator dependency-version: 13.15.22 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>

Copilot

Pull request overview

This PR implements three major changes: fixes MySQL and PostgreSQL database connectors, simplifies the UI by removing derived field functionality, and upgrades Vega from v5 to v6.

Database connector fixes: MySQL connector now uses native pymysql instead of DuckDB's MySQL extension; PostgreSQL connector improves parameter validation
UI simplification: Removes complex derived field/concept transformation features, simplifies model selection from slot-based to single selection, streamlines dialog interactions
Vega upgrade: Major version upgrade from 5.x to 6.x with corresponding dependency updates

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
`py-src/data_formulator/data_loader/mysql_data_loader.py`	Complete rewrite to use native pymysql connection instead of DuckDB extension
`py-src/data_formulator/data_loader/postgresql_data_loader.py`	Improved parameter validation and error handling
`src/components/ComponentType.tsx`	Removed `ConceptTransformation` interface and `"derived"` field source type
`src/app/dfSlice.tsx`	Simplified model selection from slot-based system to single model selection
`src/views/VisualizationView.tsx`	Changed renderer from SVG to canvas, removed derived field UI elements
`src/views/ConceptCard.tsx`	Removed derived concept editing and transformation functionality
`src/views/ModelSelectionDialog.tsx`	Simplified from multi-slot model assignment to single model selection
`src/views/DataFormulator.tsx`	Updated welcome screen layout and model testing logic
`package.json` & `yarn.lock`	Upgraded Vega from v5.32.0 to v6.2.0 and all related packages
`requirements.txt` & `pyproject.toml`	Added pymysql dependency, bumped version to 0.5.0.3
Multiple view files	Removed derived field functionality, simplified UI interactions, reduced animation durations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

py-src/data_formulator/data_loader/mysql_data_loader.py

src/views/VisualizationView.tsx

Copilot · 2025-12-04T02:19:38Z

src/views/VisualizationView.tsx

+                                        chartWidth={Math.round(config.defaultChartWidth * localScaleFactor)}
+                                        chartHeight={Math.round(config.defaultChartHeight * localScaleFactor)}
+                                        scaleFactor={1}


Scaling approach has changed significantly: Instead of applying scaleFactor post-render, it's now applied directly to width/height before rendering (lines 864-865) and scaleFactor is set to 1 (line 866). This is a substantial behavior change that could affect chart rendering quality, especially for high-DPI displays or when users adjust zoom.

The old approach scaled the rendered SVG, while the new approach pre-sizes the chart. These have different implications for resolution and rendering quality.

Suggested change

chartWidth={Math.round(config.defaultChartWidth * localScaleFactor)}

chartHeight={Math.round(config.defaultChartHeight * localScaleFactor)}

scaleFactor={1}

chartWidth={config.defaultChartWidth}

chartHeight={config.defaultChartHeight}

scaleFactor={localScaleFactor}

Copilot · 2025-12-04T02:19:38Z

src/components/ComponentType.tsx

-
-    transform?: ConceptTransformation;
-    temporary?: true; // the field is temporary, and it will be deleted unless it's saved
 }


The ConceptTransformation type and transform property have been removed from FieldItem. This is part of the larger refactoring to simplify the UI by removing derived field functionality. This is a breaking change that affects the data model significantly.

Copilot · 2025-12-04T02:19:39Z

src/views/VisualizationView.tsx

    let chartActionItems = isDataStale ? [] : (
        <Box sx={{display: "flex", flexDirection: "column", flex: 1, my: 1}}>
-            {(table.virtual || table.rows.length > 1000) && !(chartUnavailable || encodingShelfEmpty) ? (
+            {(table.virtual || table.rows.length > 5000) && !(chartUnavailable || encodingShelfEmpty) ? (


The threshold has been changed from 1000 to 5000 rows without explanation. This affects when the visualization shows a warning about large datasets. This change could impact performance or user experience for users with datasets between 1000-5000 rows.

Consider documenting the rationale for this change.

Suggested change

{(table.virtual || table.rows.length > 5000) && !(chartUnavailable || encodingShelfEmpty) ? (

{

// Threshold for showing large dataset warning changed from 1000 to 5000 rows.

// Rationale: Performance improvements and user feedback indicate that visualizations remain responsive up to 5000 rows.

// This avoids unnecessary warnings for moderately sized datasets and improves user experience.

(table.virtual || table.rows.length > 5000) && !(chartUnavailable || encodingShelfEmpty) ? (

Copilot · 2025-12-04T02:19:40Z

py-src/data_formulator/data_loader/mysql_data_loader.py

+                } for _, col_row in columns_df.iterrows()]
+
+                # Get sample data
+                sample_query = f"SELECT * FROM `{schema}`.`{table_name}` LIMIT 10"


SQL injection vulnerability: The query directly interpolates schema and table_name into SQL without proper escaping or parameterization. Even though backticks are used, this doesn't prevent SQL injection if the values contain backticks.

Use parameterized queries or properly escape table/schema names using pymysql.escape_string() or similar.

Copilot · 2025-12-04T02:19:40Z

py-src/data_formulator/data_loader/mysql_data_loader.py

+                sample_rows = json.loads(sample_df.to_json(orient="records", date_format='iso'))
+
+                # Get row count
+                count_query = f"SELECT COUNT(*) as cnt FROM `{schema}`.`{table_name}`"


SQL injection vulnerability: Same issue as above - schema and table_name are directly interpolated into the SQL query without proper escaping or parameterization.

Copilot · 2025-12-04T02:19:40Z

src/views/ModelSelectionDialog.tsx

 import CheckCircleOutlineIcon from '@mui/icons-material/CheckCircleOutline';
 import ErrorOutlineIcon from '@mui/icons-material/ErrorOutline';
 import HelpOutlineIcon from '@mui/icons-material/HelpOutline';
+import InfoOutlinedIcon from '@mui/icons-material/InfoOutlined';


Unused import InfoOutlinedIcon.

Suggested change

import InfoOutlinedIcon from '@mui/icons-material/InfoOutlined';

py-src/data_formulator/data_loader/mysql_data_loader.py

Copilot · 2025-12-04T02:19:41Z

py-src/data_formulator/data_loader/mysql_data_loader.py

+        except Exception:
+            pass


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except Exception:

pass

except Exception as e:

logger.exception("Exception occurred while closing MySQL connection in __del__")

…tor-13.15.22 Bump validator from 13.15.20 to 13.15.22

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 24 out of 26 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-05T01:51:01Z

src/app/dfSlice.tsx

-    },
-    getAllSlotTypes: () : ModelSlotType[] => {
-        return [...MODEL_SLOT_TYPES];
+        return state.models.find(m => m.id == state.selectedModelId) || state.models[0];


Incorrect equality operator: Using == for string comparison instead of ===. While JavaScript's == will work for string comparison in this case, the codebase should use strict equality === for consistency and to avoid potential type coercion issues.

Change m.id == selectedModelId to m.id === selectedModelId on line 933, and similar occurrences throughout the file.

src/views/DataFormulator.tsx

py-src/data_formulator/data_loader/mysql_data_loader.py

Copilot · 2025-12-05T01:51:02Z

py-src/data_formulator/data_loader/mysql_data_loader.py

+                sample_query = f"SELECT * FROM `{schema}`.`{table_name}` LIMIT 10"
+                sample_df = self._execute_query(sample_query)
+                sample_rows = json.loads(sample_df.to_json(orient="records", date_format='iso'))
+
+                # Get row count
+                count_query = f"SELECT COUNT(*) as cnt FROM `{schema}`.`{table_name}`"
+                count_df = self._execute_query(count_query)


SQL injection vulnerability: The query uses unsanitized schema and table names directly in f-strings without proper escaping or parameterization. Lines 181, 186 use backtick-quoted identifiers, but the values inside the backticks are not sanitized. A malicious schema/table name like `; DROP TABLE users; -- could potentially execute arbitrary SQL.

Consider using parameterized queries or proper escaping mechanisms provided by pymysql.

Copilot · 2025-12-05T01:51:02Z

py-src/data_formulator/data_loader/postgresql_data_loader.py

+            password_part = f" password={password}" if password else ""
+            attach_string = f"host={host} port={port} user={user}{password_part} dbname={database}"


Missing password sanitization in log output: The connection string is constructed with the password and could potentially be logged in error messages. On line 54, if an error occurs after this line and the attach_string is included in the error message, it would expose the password.

Consider redacting sensitive information from logs or using a more secure connection method that doesn't expose credentials in string form.

@copilot open a new pull request to apply changes based on this feedback

Copilot · 2025-12-05T01:51:03Z

py-src/data_formulator/data_loader/mysql_data_loader.py

+            self.mysql_conn = pymysql.connect(
+                host=host,
+                user=user,
+                password=password,
+                database=database,
+                port=port,
+                cursorclass=pymysql.cursors.DictCursor,
+                charset='utf8mb4'
+            )
+            self.database = database
+            logger.info(f"Successfully connected to MySQL database: {self.database}")
+        except Exception as e:
+            logger.error(f"Failed to connect to MySQL: {e}")
+            raise


Resource leak: The MySQL connection is stored in self.mysql_conn but there's no guarantee it will be closed. While close() and context manager methods are implemented (lines 264-278), there's no cleanup in the destructor (__del__), so if the object is destroyed without calling close() or using it as a context manager, the connection will leak.

Consider adding a __del__ method that calls self.close() to ensure cleanup.

@copilot open a new pull request to apply changes based on this feedback

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Add __del__ method to MySQLDataLoader for connection cleanup

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Feat/expand chart support

Add BigQuery DataLoader support

Add MongoDB Support

Mestway

Good!

dependabot bot and others added 11 commits November 17, 2025 19:54

Merge pull request #199 from microsoft/dependabot/npm_and_yarn/vega-6…

46f3852

….2.0 Bump vega from 5.33.0 to 6.2.0

Merge branch 'main' into dev

bcd47a7

Add BigQuery DataLoader

dd900ad

Merge pull request #1 from theinfinitereality/bigquery-data-loader

66690b9

Add BigQuery DataLoader

simplify ui

8ff7b6e

some cleanup

6f78c1f

fix some db connection issue

13b6877

fix and test

900ea2d

version bump

f21698a

Chenglong-MS requested a review from Copilot December 4, 2025 02:14

Copilot started reviewing on behalf of Chenglong-MS December 4, 2025 02:14 View session

Copilot finished reviewing on behalf of Chenglong-MS December 4, 2025 02:17

Copilot AI reviewed Dec 4, 2025

View reviewed changes

fix accessibility issues

8b0a523

This was referenced Dec 4, 2025

The connection to the remote MySQL keeps spinning #203

Open

Input no response for "d" in Data Connector #202

Open

Chenglong-MS and others added 6 commits December 4, 2025 17:41

small fixes

26824a4

Merge branch 'dev' into dependabot/npm_and_yarn/validator-13.15.22

bf38f06

Merge pull request #207 from microsoft/dependabot/npm_and_yarn/valida…

8fb69de

…tor-13.15.22 Bump validator from 13.15.20 to 13.15.22

Update py-src/data_formulator/data_loader/mysql_data_loader.py

0b0931a

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/views/VisualizationView.tsx

c345c70

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update py-src/data_formulator/data_loader/mysql_data_loader.py

f280be6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Chenglong-MS requested a review from Copilot December 5, 2025 01:47

Copilot started reviewing on behalf of Chenglong-MS December 5, 2025 01:47 View session

Copilot finished reviewing on behalf of Chenglong-MS December 5, 2025 01:49

Copilot AI reviewed Dec 5, 2025

View reviewed changes

Chenglong-MS and others added 2 commits December 4, 2025 18:53

Update py-src/data_formulator/data_loader/mysql_data_loader.py

72b65e7

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Initial plan

350c138

Chenglong-MS and others added 19 commits December 4, 2025 21:31

Merge branch 'dev' into copilot/sub-pr-208

0446017

Merge pull request #209 from microsoft/copilot/sub-pr-208

3624be1

Add __del__ method to MySQLDataLoader for connection cleanup

fix issues that copilot introduced, bruh

5912ff9

Expand chart type supports: Pie Chart

64d69e6

Expand chart type supports: US Map

bf477f4

simplify

5c3e801

fixes

042bf2c

Update py-src/data_formulator/data_loader/bigquery_data_loader.py

4b04973

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

changes

532fecb

Add MongoDB Support

6a53da8

Merge pull request #211 from KaranPradhan266/feat/expand-chart-support

58d86f9

Feat/expand chart support

add icon for chart templates

5f7e227

Merge pull request #206 from hurairahmateen/bigquery-data-loader

4fa8ff8

Add BigQuery DataLoader support

Merge branch 'dev' into feature/dbsupport

eee8537

Merge pull request #213 from BAIGUANGMEI/feature/dbsupport

b863916

Add MongoDB Support

some fixes with data loaders

f80a2bd

resolve dependency

6a5f0ed

fix and tested mongodb data loader

9dab794

fixes

55412f6

Chenglong-MS requested review from Mestway and danmarshall December 8, 2025 23:09

update readme and version

95e23e7

danmarshall approved these changes Dec 8, 2025

View reviewed changes

Chenglong-MS added 5 commits December 8, 2025 16:11

readme style

68ac9e6

readme style

33d8782

readme style

99ee72b

finally

57676e2

use image for cleaner readme

5939cc3

Mestway approved these changes Dec 9, 2025

View reviewed changes

Chenglong-MS merged commit 7b6fd1a into main Dec 9, 2025
7 checks passed

-            {(table.virtual || table.rows.length > 5000) && !(chartUnavailable || encodingShelfEmpty) ? (
+            {
+                // Threshold for showing large dataset warning changed from 1000 to 5000 rows.
+                // Rationale: Performance improvements and user feedback indicate that visualizations remain responsive up to 5000 rows.
+                // This avoids unnecessary warnings for moderately sized datasets and improves user experience.
+                (table.virtual || table.rows.length > 5000) && !(chartUnavailable || encodingShelfEmpty) ? (

		password_part = f" password={password}" if password else ""
		attach_string = f"host={host} port={port} user={user}{password_part} dbname={database}"

Dev #208

Dev #208

Uh oh!

Conversation

Chenglong-MS commented Dec 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Chenglong-MS Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Chenglong-MS Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Mestway left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants