WIP enable mw api emailuser backend #1498

jsnshrmn · 2025-12-16T18:56:17Z

provides features required for mass emailing via WMF Cloud VPS.

Bug: T412427

Description

Describe your changes in detail following the commit message guidelines

Rationale

Phabricator Ticket

How Has This Been Tested?

Screenshots of your changes (if appropriate):

Types of changes

What types of changes does your code introduce? Add an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Minor change (fix a typo, add a translation tag, add section to README, etc.)

@todo

- mediawiki mail backend for django - delay, retry, and maxlag handling adapted from pywikiapi - test_email - management command - email task and templates - task-specific mail backend switching - additional config - optional .env # @todo: document Bug: T409420 Bug: T412427

Bug: T409420

- new email message model - proxy djmail messages and add model manager with helper methods - cleanup_email cmd deletes draft emails with subject and optionally clears user flag - requires email subject - optional argument for userprofile flag field to clear sent status Bug: T409420 Bug: T412427

- limit emails sent per run to 1000 by default - naively creates one session + login per email email mediawiki backend: - skip accounts that share an email address Bug: T409420 Bug: T412427

Bug: T409420 Bug: T412427

email tasks: create and save djmail message objects mediawiki backend: raise exceptions when mail can't be sent so status is tracked in djmail objects survey_active_users: clean up email sending Bug: T409420 Bug: T412427

jsnshrmn · 2025-12-19T21:07:24Z

TWLight/emails/management/commands/cleanup_email.py

+from TWLight.emails.models import Message
+
+
+class Command(BaseCommand):


note that this is simple db lookups, so the command would have to be run multiple times with each translated subject line to catch all the variations, eg.
python manage.py cleanup_email --subject "The Wikipedia Library needs your help!" --userprofile_flag_field "survey_email_sent"

would just get the english emails

jsnshrmn · 2025-12-19T21:09:13Z

TWLight/emails/models.py

+from django.contrib.auth.models import User
+from django.db import models
+
+from djmail.models import Message as DjMail


this is creating a dependency issue. I just ran out of time before I could look at it. There are usually methods for using a string representation of the class to resolve this kind of issue, sometimes in a model, sometimes in app config, or maybe this needs an app label and a local migration added to define the dependency.

jsnshrmn · 2025-12-19T21:10:23Z

TWLight/emails/tasks.py

    Any time the related managment command is run, this sends a survey
    invitation to qualifying editors.
    """
+    connection = (


If a connection isn't passed in, this will create 1 connection for each email. Bad for bulk mail, but fine for transactional mail.

jsnshrmn · 2025-12-19T21:12:21Z

TWLight/emails/tasks.py

+    model_instance.save()
+
+    # send email and update state in djmail
+    _safe_send_message(model_instance, connection)


Obviously, this is an internal method. Forking djmail would be cleaner, but this works for now.

jsnshrmn · 2025-12-19T21:14:38Z

TWLight/users/management/commands/survey_active_users.py

                # have an non-wikimedia.org email address
-                Q(email__isnull=False) & ~Q(email__endswith="@wikimedia.org"),
+                Q(email__isnull=False)
+                & ~Q(email="")


I realized we were catching folks with blank email address fields

jsnshrmn · 2025-12-19T21:20:02Z

TWLight/users/management/commands/survey_active_users.py

+        connection.close()
+
+        # Record that we sent the email so that we only send one.
+        # user.userprofile.survey_email_sent = True


I commented this out for repeated local testing, but it's kind of bad anyway. If the djmail message created for this has a status of "draft", then the user wasn't actually sent the email. We're having to go check that manually for the cleanup command anyway.

I was thinking of just dropping the flag altogether and just switching to doing a lookup based on email address & subject, or adding a m2m user:message relationship, etc. I might recommend starting with a lookup and then only switching to a relationship if we really fork the middleware. The proxy model has some helper methods on it that you could use as inspiration for making this better.

jsnshrmn · 2025-12-19T21:21:01Z

docker-compose.override.yml

 services:
  twlight:
+    extra_hosts:
+      - "host.docker.internal:host-gateway"


needed to be able to talk to a separate mediawiki container

jsnshrmn · 2025-12-19T21:21:41Z

docker-compose.override.yml

-      - ./conf/local.twlight.env
+      - path: ./conf/local.twlight.env
+        required: true
+      - path: .env


This is where you can stick your bot credential env vars

jsnshrmn · 2025-12-19T21:22:11Z

conf/local.twlight.env

 DJANGO_DB_HOST=db
-DJANGO_EMAIL_BACKEND=django_dkim.backends.console.EmailBackend
+#DJANGO_EMAIL_BACKEND=TWLight.emails.backends.mediawiki.EmailBackend
+DJANGO_EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend


you can swap this out to push everything through the mediawiki emailuser api.

jsnshrmn · 2025-12-19T21:24:17Z

TWLight/emails/backends/mediawiki.py

+                    )
+                emailable_data = _json_maxlag(emailable_response)
+                emailable = "emailable" in emailable_data["query"]["users"][0]
+                if not emailable:


in mediawiki you'll need to add and confirm email addresses for users with wp_usernames that match what's in your TWLight. It's kind of a pain.

jsnshrmn had a problem deploying to cicd December 16, 2025 18:56 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from f685cf3 to 0201ab9 Compare December 16, 2025 18:59

jsnshrmn temporarily deployed to cicd December 16, 2025 18:59 — with GitHub Actions Inactive

jsnshrmn temporarily deployed to cicd December 17, 2025 19:32 — with GitHub Actions Inactive

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from 065941a to c086792 Compare December 17, 2025 19:45

jsnshrmn had a problem deploying to cicd December 17, 2025 19:46 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from c086792 to e6fd993 Compare December 18, 2025 21:04

jsnshrmn had a problem deploying to cicd December 18, 2025 21:04 — with GitHub Actions Failure

jsnshrmn changed the title ~~WIP requirements: use twl djmail fork~~ WIP enable mw api emailuser backend Dec 18, 2025

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from e6fd993 to 1608291 Compare December 18, 2025 23:09

jsnshrmn had a problem deploying to cicd December 18, 2025 23:09 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from 1608291 to 1bc7bb9 Compare December 18, 2025 23:22

jsnshrmn had a problem deploying to cicd December 18, 2025 23:23 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from 1bc7bb9 to a61018e Compare December 19, 2025 04:30

jsnshrmn temporarily deployed to cicd December 19, 2025 04:30 — with GitHub Actions Inactive

jsnshrmn added 2 commits December 19, 2025 11:20

users: add survey email to admin as non-editable

6a9bbd7

Bug: T409420

user survey email cmd: exclude blank email

ff8b470

Bug: T409420

jsnshrmn had a problem deploying to cicd December 19, 2025 18:22 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from 937980a to 66d0f18 Compare December 19, 2025 18:34

jsnshrmn had a problem deploying to cicd December 19, 2025 18:34 — with GitHub Actions Failure

jsnshrmn had a problem deploying to cicd December 19, 2025 19:03 — with GitHub Actions Failure

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from b0b158c to 289f1a1 Compare December 19, 2025 19:20

jsnshrmn had a problem deploying to cicd December 19, 2025 19:20 — with GitHub Actions Failure

jsnshrmn added 2 commits December 19, 2025 13:38

user survey email cmd: use mw email backend

54611ec

- limit emails sent per run to 1000 by default - naively creates one session + login per email email mediawiki backend: - skip accounts that share an email address Bug: T409420 Bug: T412427

user survey email cmd: 1 session per batch

6ffba2d

Bug: T409420 Bug: T412427

jsnshrmn force-pushed the Jsn.sherman/T412427 branch from 289f1a1 to 6ffba2d Compare December 19, 2025 19:56

jsnshrmn had a problem deploying to cicd December 19, 2025 19:56 — with GitHub Actions Failure

user survey email cmd: track emails in djmail

631a050

email tasks: create and save djmail message objects mediawiki backend: raise exceptions when mail can't be sent so status is tracked in djmail objects survey_active_users: clean up email sending Bug: T409420 Bug: T412427

jsnshrmn had a problem deploying to cicd December 19, 2025 20:49 — with GitHub Actions Failure

jsnshrmn commented Dec 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP enable mw api emailuser backend #1498

WIP enable mw api emailuser backend #1498

Uh oh!

jsnshrmn commented Dec 16, 2025 •

edited

Loading

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

jsnshrmn Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		from TWLight.emails.models import Message


		class Command(BaseCommand):

WIP enable mw api emailuser backend #1498

Are you sure you want to change the base?

WIP enable mw api emailuser backend #1498

Uh oh!

Conversation

jsnshrmn commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Rationale

Phabricator Ticket

How Has This Been Tested?

Screenshots of your changes (if appropriate):

Types of changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jsnshrmn commented Dec 16, 2025 •

edited

Loading