Add Alt Text Generation experiment #156

Jameswlepage · 2025-12-19T18:00:28Z

Addresses #44

Summary

Adds AI-powered alt text generation for images using vision models:

Block Editor integration: Adds "Generate Alt Text" button to Image block's InspectorControls
Media Library integration: Works in upload.php and media modals throughout WordPress
Flexible input: Supports attachment IDs, URLs, and data URIs
Smart processing: Converts images to data URIs for AI analysis, handles local and remote files
Quality output: Generates concise, descriptive alt text (max 125 characters)

Implementation Details

Uses ai/alt-text-generation ability with vision models (Claude Haiku, GPT-5 Nano, Gemini Flash)
Block editor script hooks into editor.BlockEdit filter
Media script integrates with WordPress media frame attachment details
Includes shared run-ability.ts utility for Abilities API with REST fallback

Test plan

Enable experiment under Settings → AI Experiments
Open block editor, add an Image block, and click "Generate Alt Text"
Visit Media Library (upload.php), select an image, and test alt text generation
Test with images that have no alt text set
Verify generated text is under 125 characters and descriptive

Adds AI-powered alt text generation for images using vision models: - Block editor integration via InspectorControls for Image blocks - Media Library integration for upload.php and media modals - Supports attachment IDs, URLs, and data URIs as input - Uses AI vision models (Claude, GPT, Gemini) to analyze images - Generates concise, descriptive alt text (max 125 chars) Includes shared run-ability.ts utility for Abilities API fallback.

github-actions · 2025-12-19T18:00:38Z

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @andreizanik, @hellobertha.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: andreizanik, hellobertha.

Co-authored-by: Jameswlepage <isotropic@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: theaminulai <theaminuldev@git.wordpress.org>
Co-authored-by: karmatosed <karmatosed@git.wordpress.org>
Co-authored-by: joedolson <joedolson@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

codecov · 2025-12-19T18:03:39Z

Codecov Report

❌ Patch coverage is 40.12945% with 185 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.45%. Comparing base (fd0d510) to head (1b42e4b).
⚠️ Report is 41 commits behind head on develop.

Files with missing lines	Patch %	Lines
includes/Abilities/Image/Alt_Text_Generation.php	48.14%	112 Missing ⚠️
...iments/Alt_Text_Generation/Alt_Text_Generation.php	6.41%	73 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #156      +/-   ##
=============================================
- Coverage      50.33%   49.45%   -0.89%     
- Complexity       366      463      +97     
=============================================
  Files             26       29       +3     
  Lines           1951     2283     +332     
=============================================
+ Hits             982     1129     +147     
- Misses           969     1154     +185

Flag	Coverage Δ
unit	`49.45% <40.12%> (-0.89%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR adds AI-powered alt text generation capabilities for images across WordPress's block editor and media library interfaces using vision-capable AI models.

Key Changes:

Integrates alt text generation into the Image block via InspectorControls with generate/regenerate functionality
Adds media library support for upload.php, media modals, and attachment edit screens through DOM manipulation
Implements a shared runAbility utility that gracefully falls back from Abilities API to REST calls

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
webpack.config.js	Adds two webpack entry points for the block editor and media library scripts
src/utils/run-ability.ts	Implements shared utility for executing abilities with fallback to REST API
src/experiments/alt-text-generation/types.ts	Defines TypeScript interfaces for Image block attributes
src/experiments/alt-text-generation/media.ts	Implements DOM-based alt text generation for media library and modals
src/experiments/alt-text-generation/index.tsx	Registers block editor filter to inject controls into Image blocks
src/experiments/alt-text-generation/components/AltTextControls.tsx	React component providing UI for alt text generation in block inspector
includes/Experiments/Alt_Text_Generation/Alt_Text_Generation.php	Experiment registration with asset enqueuing hooks for editor and media contexts
includes/Experiment_Loader.php	Registers the new experiment in the experiments array
includes/Abilities/Alt_Text_Generation/system-instruction.php	System prompt defining alt text generation requirements (concise, descriptive, accessible)
includes/Abilities/Alt_Text_Generation/Alt_Text_Generation.php	Core ability implementation handling image processing, data URI conversion, and AI prompting
docs/experiments/alt-text-generation.md	Comprehensive documentation of hooks, data flow, testing steps, and implementation notes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/experiments/alt-text-generation/components/AltTextControls.tsx

src/utils/run-ability.ts

Copilot · 2025-12-19T18:09:56Z

includes/Abilities/Image/Alt_Text_Generation.php

+			return null;
+		}
+
+		$contents = file_get_contents( $file_path ); // phpcs:ignore WordPress.WP.AlternativeFunctions.file_get_contents_file_get_contents


Using 'file_get_contents' to read potentially large image files could cause memory issues. Consider checking file size first or using streaming approaches for larger files to prevent memory exhaustion.

src/experiments/alt-text-generation/index.tsx

src/utils/run-ability.ts

src/experiments/alt-text-generation/media.ts

includes/Abilities/Image/Alt_Text_Generation.php

Copilot · 2025-12-19T18:10:00Z

includes/Abilities/Image/Alt_Text_Generation.php

+	protected function download_remote_image_to_temp_file( string $url ) {
+		if ( ! function_exists( 'download_url' ) ) {
+			require_once ABSPATH . 'wp-admin/includes/file.php';
+		}
+
+		$temp_file = download_url( $url );
+


The download_remote_image_to_temp_file method passes the untrusted image_url directly into download_url, which will make an outbound HTTP request from the server to any host the caller supplies. A user with the upload_files capability can invoke the ai/alt-text-generation REST ability and use this to perform server-side requests to arbitrary (including internal) URLs, which is an SSRF risk even though the response body is only used as an image file. To reduce this risk, restrict outbound requests to a well-defined set of trusted hosts or validate the URL against private/local address ranges before calling download_url.

jeffpaul · 2025-12-20T18:32:59Z

@Jameswlepage for the overlap between this experiment and the image generation one, perhaps we default the image gen prompt as the alt text there but still present the button to generate alternate alt text ;) for those images?

jeffpaul · 2025-12-20T19:27:57Z

@Jameswlepage does this actually close #44 or is there still more to add to fully cover the functionality from that issue?

jeffpaul · 2025-12-20T20:40:05Z

@Jameswlepage can you add tests here like was done in https://github.com/WordPress/ai/pull/147/changes#diff-0cf348eda48cae411ad9d44a2e5b449603a971f2313b0a849893ff4cf8280c46?

jeffpaul · 2025-12-20T21:17:53Z

Reminder for us to add a screenshot gif and entry in the readme.txt for this feature before merging/releasing.

includes/Abilities/Image/alt-text-system-instruction.php

…g patterns in other files

…e existing WordPress hooks to add that button in. Adjust our JS accordingly with this new setup.

…tton and ensure this works properly

…ply better protections around path traversal

Jameswlepage requested review from Copilot and dkotter and removed request for Copilot December 19, 2025 18:02

Copilot started reviewing on behalf of Jameswlepage December 19, 2025 18:03 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

Jameswlepage marked this pull request as draft December 19, 2025 18:42

Merge branch 'develop' into feature/alt-text-generation

42bae60

jeffpaul assigned Jameswlepage and JasonTheAdams and unassigned JasonTheAdams Dec 20, 2025

jeffpaul requested review from JasonTheAdams and felixarntz December 20, 2025 18:27

jeffpaul added this to WordPress AI Planning & Roadmap Dec 20, 2025

jeffpaul added this to the 0.2.0 milestone Dec 20, 2025

jeffpaul moved this to Needs review in WordPress AI Planning & Roadmap Dec 20, 2025

JasonTheAdams removed their request for review December 21, 2025 05:59

andreizanik reviewed Dec 22, 2025

View reviewed changes

includes/Abilities/Image/alt-text-system-instruction.php Outdated Show resolved Hide resolved

jeffpaul linked an issue Jan 7, 2026 that may be closed by this pull request

Alt Text Generation #44

Open

jeffpaul removed this from WordPress AI Planning & Roadmap Jan 7, 2026

jeffpaul modified the milestones: 0.2.0, 0.3.0 Jan 13, 2026

Merge branch 'develop' into feature/alt-text-generation

f5e5cb6

dkotter added 14 commits January 28, 2026 10:29

Move the Ability to the Image directory. Minor changes to match codin…

ea83864

…g patterns in other files

Add function to get preferred vision models

8783bc6

Load the proper Ability. Minor code improvements

05bd9d7

Add missing dependencies

a8b014b

Fix linting and typescript errors

04d7141

Switch from fully injecting our generate button via JS and instead us…

20f681d

…e existing WordPress hooks to add that button in. Adjust our JS accordingly with this new setup.

Add a meta box to the attachment edit screen that has the generate bu…

fd99527

…tton and ensure this works properly

Update docs

6a86fc6

Add unit tests

68adddd

Fix filter name

3b05a50

Add E2E tests

61c9465

Fix linting errors

8f6a7d2

Update system instructions

150ca22

Apply code review suggestions. Switch from strlen to mb_strlen and ap…

1b42e4b

…ply better protections around path traversal

dkotter marked this pull request as ready for review January 28, 2026 23:41

dkotter self-assigned this Jan 28, 2026

Add Alt Text Generation experiment #156

Are you sure you want to change the base?

Add Alt Text Generation experiment #156

Uh oh!

Conversation

Jameswlepage commented Dec 19, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation Details

Test plan

Uh oh!

github-actions bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unlinked Accounts

Uh oh!

codecov bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

jeffpaul commented Dec 20, 2025

Uh oh!

jeffpaul commented Dec 20, 2025

Uh oh!

jeffpaul commented Dec 20, 2025

Uh oh!

jeffpaul commented Dec 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Jameswlepage commented Dec 19, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Dec 19, 2025 •

edited

Loading

codecov bot commented Dec 19, 2025 •

edited

Loading