Implement alerts for performance UI test results #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

pwrightcertinia merged 34 commits into apex-dev-tools:main from rjaiswalcertinia:main

Oct 28, 2025

CHANGELOG.md

-Original file line number
+Diff line change
@@ -1,5 +1,11 @@
     # Benchmarker - Changelog
+    ## 6.0.0
+    ### Breaking Changes
+    - Database schema updated: Use `db/migrations/V6__create_performance_ui_alert_table.sql` script to create new table to store alerts for performance UI test results.
     ## 5.0.0
     ### Breaking Changes
@@ Expand Down @@

db/init.sql

-Original file line number
+Diff line change
@@ Expand Up / @@ -100,4 +100,15 @@ CREATE TABLE IF NOT EXISTS performance.ui_test_result ( @@
         component_load_time integer,
         salesforce_load_time integer,
         overall_load_time integer
+    );
+    CREATE TABLE IF NOT EXISTS performance.ui_alert (
+        id serial PRIMARY KEY,
+        create_date_time timestamp without time zone DEFAULT now() NOT NULL,
+        update_date_time timestamp without time zone DEFAULT now() NOT NULL,
+        ui_test_result_id integer,
+        alert_type text,
+        test_suite_name text,
+        individual_test_name text,
+        component_load_time_degraded integer
     );

db/migrations/V6__create_performance_ui_alert_table.sql

-Original file line number
+Diff line change
@@ -0,0 +1,10 @@
+    CREATE TABLE IF NOT EXISTS performance.ui_alert (
+        id serial PRIMARY KEY,
+        create_date_time timestamp without time zone DEFAULT now() NOT NULL,
+        update_date_time timestamp without time zone DEFAULT now() NOT NULL,
+        ui_test_result_id integer,
+        alert_type text,
+        test_suite_name text,
+        individual_test_name text,
+        component_load_time_degraded integer
+    );

docs/user/alerts.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -110,3 +110,48 @@ await TransactionProcess.executeTestStep(..., alertInfo); @@
     ```txt
     Note: If the test level threshold is misconfigured below the average, you get an alert with a value of 0. Recommend filtering out zero alerts when querying for new records.
     ```
+    # UI Alerts
+    UI Alerts can be stored in the database to monitor performance degradation over time. Each record’s degradation value represents the difference between the average of the first 5 ui test results and the average of the subsequent 10 ui test results recorded within the past 30 days. A minimum of 15 results within a 30-day window is required to trigger UI alert storage.
+    ## Usage
+    ### Environment Variables
+    To enable UI alerts globally, set the `STORE_UI_ALERTS` variable to `true` in the environment file
+    To override the default normal component load time threshold (1000 ms), set the `NORMAL_COMPONENT_LOAD_THRESHOLD` variable to any desired value in the environment file
+    To override the default critical component load time threshold (10000 ms), set the `CRITICAL_COMPONENT_LOAD_THRESHOLD` variable to any desired value in the environment file
+    #### Example Scenario
+    * The average component load time for the first 5 results was 1500 ms, while the next 10 results averaged 1000 ms.
+    * You will get an normal alert, saying it has degraded by 500 above the average.
+    ### Test Level Thresholds
+    Alternatively, thresholds can be configured at the test level. If the difference between the average of the first 5 and the subsequent 10 ui test results exceeds the defined custom threshold, an alert will be stored. The degradation value remains the difference between the average of the first 5 results and the average of the next 10 results recorded. The threshold simply determines whether an alert is triggered.
+    ```ts
+    // Replace global alert behaviour with exact thresholds
+    const customThresholds: UiAlertThresholds = new UiAlertThresholds();
+    customThresholds.componentLoadTimeThresholdNormal = 50;
+    customThresholds.componentLoadTimeThresholdCritical = 100;
+    // Enable alerting for this test if not already active
+    const alertInfo: UiAlertInfo = new UiAlertInfo();
+    alertInfo.storeAlerts = true;
+    alertInfo.uiAlertThresholds = customThresholds;
+    const testResult: UiTestResultDTO = {
+            ...,
+            alertInfo,
+          };
+    await saveUiTestResult([testResult]);
+    ```
+    #### Example Scenario
+    * Test-level thresholds for normal and critical component load times are set to 50 ms and 100 ms, respectively. The average load time for the first 5 results was 1500 ms, while the subsequent 10 results averaged 1000 ms.
+    * You will get an critical alert, saying it has degraded by 500 above the average.

src/database/connection.ts

-Original file line number
+Diff line change
@@ Expand Up / @@ -12,6 +12,7 @@ import { ExecutionInfo } from './entity/execution'; @@
     import { Alert } from './entity/alert';
     import { TestInfo } from './entity/testInfo';
     import { UiTestResult } from './entity/uiTestResult';
+    import { UiAlert } from './entity/uiAlert';
     export const DB_ENTITIES = [
       TestResult,
@@ Expand All / @@ -21,6 +22,7 @@ export const DB_ENTITIES = [ @@
       Alert,
       TestInfo,
       UiTestResult,
+      UiAlert,
     ];
     let connection: DataSource | null = null;
@@ Expand Down @@

src/database/entity/uiAlert.ts

-Original file line number
+Diff line change
@@ -0,0 +1,34 @@
+    /** @ignore */
+    /*
+     * Copyright (c) 2025 Certinia Inc. All rights reserved.
+     */
+    import { Entity, Column } from 'typeorm';
+    import { PerformanceBaseEntity } from './base';
+    import {
+      DEFAULT_NUMERIC_VALUE,
+      DEFAULT_STRING_VALUE,
+    } from '../../shared/constants';
+    @Entity({ name: 'ui_alert' })
+    export class UiAlert extends PerformanceBaseEntity {
+      [key: string]: number | string | Date | boolean | undefined;
+      @Column('integer', { nullable: true, name: 'ui_test_result_id' })
+      public uiTestResultId = DEFAULT_NUMERIC_VALUE;
+      @Column('text', { nullable: true, name: 'test_suite_name' })
+      public testSuiteName = DEFAULT_STRING_VALUE;
+      @Column('text', { nullable: true, name: 'individual_test_name' })
+      public individualTestName = DEFAULT_STRING_VALUE;
+      @Column('text', { nullable: true, name: 'alert_type' })
+      public alertType = DEFAULT_STRING_VALUE;
+      @Column('integer', { nullable: true, name: 'component_load_time_degraded' })
+      public componentLoadTimeDegraded = DEFAULT_NUMERIC_VALUE;
+      public constructor() {
+        super();
+      }
+    }

src/database/uiAlertInfo.ts

-Original file line number
+Diff line change
@@ -0,0 +1,93 @@
+    /** @ignore */
+    /**
+     * Copyright (c) 2025 Certinia, Inc. All rights reserved.
+     */
+    import { getConnection } from './connection';
+    import { UiAlert } from './entity/uiAlert';
+    import { UiTestResult } from './entity/uiTestResult';
+    export async function saveAlerts(
+      testResultsDB: UiTestResult[],
+      alerts: UiAlert[]
+    ) {
+      alerts.forEach(alert => {
+        const match = testResultsDB.find(
+          result =>
+            result.testSuiteName === alert.testSuiteName &&
+            result.individualTestName === alert.individualTestName
+        );
+        if (match) {
+          alert.uiTestResultId = match.id;
+        }
+      });
+      const connection = await getConnection();
+      return connection.manager.save(alerts);
+    }
+    export async function getAverageLimitValuesFromDB(
+      suiteAndTestNamePairs: { testSuiteName: string; individualTestName: string }[]
+    ) {
+      const connection = await getConnection();
+      const suiteAndTestNameConditions = suiteAndTestNamePairs
+        .map(pair => `('${pair.testSuiteName}', '${pair.individualTestName}')`)
+        .join(', ');
+      const query = `
+        WITH ranked AS (
+            SELECT
+                test_suite_name,
+                individual_test_name,
+                component_load_time,
+                create_date_time,
+                ROW_NUMBER() OVER (
+                    PARTITION BY test_suite_name, individual_test_name
+                    ORDER BY create_date_time DESC
+                ) AS rn
+            FROM performance.ui_test_result
+            WHERE (create_date_time >= CURRENT_TIMESTAMP - INTERVAL '30 DAYS')
+              AND (test_suite_name, individual_test_name) IN (${suiteAndTestNameConditions})
+        )
+        SELECT
+            test_suite_name,
+            individual_test_name,
+            ROUND(AVG(CASE WHEN rn BETWEEN 1 AND 5 THEN component_load_time END)::numeric, 0) AS avg_first_5,
+            ROUND(AVG(CASE WHEN rn BETWEEN 6 AND 15 THEN component_load_time END)::numeric, 0) AS avg_next_10
+        FROM ranked
+        GROUP BY test_suite_name, individual_test_name
+        HAVING COUNT(*) >= 15
+      `;
+      const resultsMap: {
+        [key: string]: {
+          avg_first_5: number;
+          avg_next_10: number;
+        };
+      } = {};
+      try {
+        const result = await connection.query(query);
+        // Populate the results map
+        result.forEach(
+          (row: {
+            test_suite_name: string;
+            individual_test_name: string;
+            avg_first_5: number;
+            avg_next_10: number;
+          }) => {
+            const key = `${row.test_suite_name}_${row.individual_test_name}`;
+            resultsMap[key] = {
+              avg_first_5: row.avg_first_5 ?? 0,
+              avg_next_10: row.avg_next_10 ?? 0,
+            };
+          }
+        );
+        return resultsMap;
+      } catch (error) {
+        console.error('Error in fetching the average values: ', error);
+        return {};
+      }
+    }

src/database/uiTestResult.ts

-Original file line number
+Diff line change
@@ Expand Up / @@ -6,6 +6,29 @@ import { UiTestResult } from './entity/uiTestResult'; @@
     import { saveRecords } from './saveRecords';
     import { getConnection } from './connection';
     import { MoreThanOrEqual } from 'typeorm';
+    import { generateValidAlerts } from '../services/result/uiAlert';
+    import { saveAlerts } from './uiAlertInfo';
+    /**
+     * Describes the Thresholds for different limits
+     * Use this class to specify thresholds such as: componentLoadTimeThresholdNormal and componentLoadTimeThresholdCritical.
+     */
+    export class UiAlertThresholds {
+      componentLoadTimeThresholdNormal: number;
+      componentLoadTimeThresholdCritical: number;
+    }
+    export class UiAlertInfo {
+      /**
+       * Describes whether alerts need to be stored or not at the test level
+       */
+      public storeAlerts: boolean;
+      /**
+       * Defines custom thresholds at the test level. When specified, these values take precedence over defaults and those set in the environment file.
+       */
+      public uiAlertThresholds: UiAlertThresholds;
+    }
     /**
      * Data Transfer Object for UI Test Results
@@ Expand All / @@ -16,6 +39,7 @@ export interface UiTestResultDTO { @@
       componentLoadTime?: number;
       salesforceLoadTime?: number;
       overallLoadTime: number;
+      alertInfo?: UiAlertInfo;
     }
     /**
@@ Expand Down Expand Up / @@ -59,6 +83,10 @@ export async function saveUiTestResult( @@
     ): Promise<UiTestResultDTO[]> {
       const entities = testStepResults.map(dtoToEntity);
       const savedEntities = await saveRecords<UiTestResult>(entities);
+      const alerts = await generateValidAlerts(testStepResults);
+      if (alerts.length) {
+        await saveAlerts(savedEntities, alerts);
+      }
       return savedEntities.map(entityToDto);
     }
@@ Expand Down @@

src/index.ts

-Original file line number
+Diff line change
@@ Expand Up / @@ -109,6 +109,8 @@ export { @@
       loadUiTestResults,
       UiTestResultDTO,
       UiTestResultFilterOptions,
+      UiAlertInfo,
+      UiAlertThresholds,
     } from './database/uiTestResult';
     export namespace Constants {
@@ Expand Down @@

src/services/defaultRanges.ts

-Original file line number
+Diff line change
@@ Expand Up / @@ -6,16 +6,14 @@ import { RangeCollection } from './ranges'; @@
     export const DEFAULT_RANGES: RangeCollection = {
       dml_ranges: [
-        { start_range: 0, end_range: 50, offset_threshold: 5 },
-        { start_range: 51, end_range: 100, offset_threshold: 3 },
-        { start_range: 101, end_range: 120, offset_threshold: 2 },
-        { start_range: 121, end_range: 149, offset_threshold: 1 },
+        { start_range: 0, end_range: 50, offset_threshold: 3 },
+        { start_range: 51, end_range: 100, offset_threshold: 2 },
+        { start_range: 101, end_range: 149, offset_threshold: 1 },
       ],
       soql_ranges: [
-        { start_range: 0, end_range: 50, offset_threshold: 10 },
-        { start_range: 51, end_range: 80, offset_threshold: 5 },
-        { start_range: 81, end_range: 90, offset_threshold: 2 },
-        { start_range: 91, end_range: 99, offset_threshold: 1 },
+        { start_range: 0, end_range: 30, offset_threshold: 3 },
+        { start_range: 31, end_range: 60, offset_threshold: 2 },
+        { start_range: 61, end_range: 99, offset_threshold: 1 },
       ],
       cpu_ranges: [
         { start_range: 0, end_range: 2000, offset_threshold: 3000 },
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement alerts for performance UI test results #53

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!