Update sequence diagrams for sending submission emails

We no longer use GOV.UK Notify to send submission emails, instead we use
Amazon SES asynchronously. Our documentation and sequence diagrams need
to be updated to reflect this, this commit:

- Add Amazon SES to high level architecture diagram
- Removes references to GOV.UK Notify from sequence diagram for filling
  in a form
- Moves sequence diagrams related to sending submission emails from file
  upload file to a new file
- Updates details around updating submission record with delivery status
  and time

Co-authored-by: Stephen Daly <stephen.daly@digital.cabinet-office.gov.uk>

Author: lfdebrux

diagrams/sequence-diagrams/README.md

@@ -38,15 +38,15 @@ graph TD
     filler --- runner --- admin
 
     notify(GOV.UK Notify)
+    ses(Amazon Simple Email Service)
 
     pay(GOV.UK Pay)
     filler -.- pay -.- runner
 
     filler -.- notify
-    notify --- runner
-    processor --- notify
-
-
+    notify -.- runner
+    ses --- runner
+    processor --- ses
 ```
 
 ## Links to Sequence Diagrams

diagrams/sequence-diagrams/file-upload.md

@@ -138,157 +138,3 @@ runner->>browser: HTTP 302 redirect response
 
 note over user,runner: User completes the rest of the questions
 ```
-
-### User submits their form
-```mermaid
-
----
-title: User submits their form
----
-
-sequenceDiagram
-
-autonumber
-
-actor user as User
-
-participant browser as Browser
-
-participant runner as Forms Runner
-participant runner-db as Forms Runner database
-participant solidqueue-db as Solid Queue database
-
-browser->>runner: GET check your answers
-runner->>browser: HTTP 200 response
-browser->>user: display check your answers page
-
-user->>browser: select "Submit"
-browser->>runner: POST submit form
-runner->>runner-db: save Submission
-runner->>solidqueue-db: enqueue send submission job
-
-runner->>browser: HTTP 302 response
-browser->>runner: GET confirmation page
-runner->>browser: HTTP 200 response
-browser->>user: Display confirmation page
-
-```
-
-### Sending the submission email asynchronously
-
-```mermaid
-
----
-title: Sending the submission email asynchronously 
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Forms runner worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant s3 as Amazon S3
-participant ses as Amazon SES
-participant sns as Amazon SNS
-participant inbox as Email inbox
-participant sentry as Sentry
-
-actor processor
-
-worker->>solidqueue-db: dequeue send submission job
-worker->>runner-db: get Submission
-worker->>s3: get file(s)
-worker->>ses: send email
-
-break error 
-  alt AWS SDK error and max retries not reached
-    worker->>solidqueue-db: schedule retry
-  else
-    worker->>sentry: send error
-  end
-end
-
-ses->>worker: return message_id
-worker->>runner-db: set mail_message_id on Submission
-worker->>runner-db: update mail_status of Submission to "pending"
-
-ses-)inbox: send email
-note over ses,inbox: happens some time later
-
-alt email sent successfully
-  ses->>sns: send delivery notification
-  processor->>inbox: get form from inbox
-  processor->>processor: process form
-else email bounces
-  ses->>sns: send bounce notification
-  note over ses,sns: We have an SQS queue subscribed to the<br/> SNS topic and a recurring task to poll<br/>the SQS queue.
-end
-
-worker->>worker: Recurring task deletes Submissions<br/>that haven't bounced after 7 days
-```
-
-### Handling email bounces/complaints
-```mermaid
-
----
-title: Handling email bounces/complaints
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Solid Queue worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant sqs as Amazon SQS
-participant sentry as Sentry
-
-actor support as Forms team tech support
-
-worker->>solidqueue-db: enqueue recurring receive bounces job
-worker->>solidqueue-db: dequeue receive bounces job
-worker->>sqs: get messages from bounces and complaints queue
-alt there is a bounce SQS message
-  worker->>runner-db: get Submission by the message_id in the SQS message
-  worker->>runner-db: update mail_status of Submission to "bounced"
-  worker->>worker: Log with the submission details
-  worker->>sentry: send error event
-  sentry->>support: Alert via Slack
-  support->>support: Identify why the email bounced
-  support->>support: Run rake task to retry submission
-else there is a complaint SQS message
-  worker->>runner-db: get Submission by the message_id in the SQS message
-  worker->>worker: Log with the submission details
-end
-
-```
-
-### Handling successful deliveries
-
-```mermaid
-
----
-title: Handling successful deliveries
----
-
-sequenceDiagram
-
-autonumber
-
-participant worker as Solid Queue worker
-participant solidqueue-db as Solid Queue database
-participant runner-db as Forms Runner database
-participant sqs as Amazon SQS
-
-actor support as Forms team tech support
-
-worker->>solidqueue-db: enqueue recurring receive deliveries job
-worker->>solidqueue-db: dequeue receive deliveries job
-worker->>sqs: get messages from deliveries queue
-worker->>runner-db: get Submission by the message_id in the SQS message
-worker->>runner-db: update mail_status of Submission to "delivered"
-note over worker,runner-db: we don't currently use the "delivered" status for anything other than for information
-```

diagrams/sequence-diagrams/filling-in-a-form.md

@@ -7,7 +7,6 @@ sequenceDiagram
   participant govuk as GOV.UK website
   participant runner as forms-runner
   participant admin as forms-admin
-  participant notify as GOV.UK Notify
   participant inbox as shared email inbox
   actor processor as Form processor
 
@@ -41,13 +40,13 @@ sequenceDiagram
   user->>browser: Click "Submit"
   browser->>runner: POST /form/{form id}/{form slug}/submit-answers
   runner->>admin: GET /api/v2/forms/{form id}/live
-  runner->>notify: client.send_email()
-  runner->>runner: Delete user session
+  runner->>runner: save Submission
+  runner->>runner: enqueue send submission job
   runner-->>browser: 302
   browser->>runner: GET /form/{form id}/{form slug}/submitted
   runner->>admin: GET /api/v2/forms/{form id}/live
   browser-->>user: Show confirmation page
-  notify->>inbox: Send email
+  runner->>inbox: Send email
   inbox->>processor: Read email
 ```
 
@@ -61,7 +60,6 @@ sequenceDiagram
   participant browser as Web Browser
   participant runner as forms-runner
   participant admin as forms-admin
-  participant notify as GOV.UK Notify
 
   user->>browser: Click link to form
   browser->>runner: GET /form/{form id}/{form slug}
@@ -75,8 +73,6 @@ sequenceDiagram
   end
 ```
 
-
-
 ## Brief summary
 
 The users answers are store in Redis in the following structure as they progress through the form.

diagrams/sequence-diagrams/sending-submission-emails.md

@@ -0,0 +1,158 @@
+# Sending submission emails asynchronously
+
+forms-runner uses SolidQueue and Amazon SES/SNS to send submissions asynchronously.
+
+## User submits their form
+
+```mermaid
+
+---
+title: User submits their form
+---
+
+sequenceDiagram
+
+autonumber
+
+actor user as User
+
+participant browser as Browser
+
+participant runner as Forms Runner
+participant runner-db as Forms Runner database
+participant solidqueue-db as Solid Queue database
+
+browser->>runner: GET check your answers
+runner->>browser: HTTP 200 response
+browser->>user: display check your answers page
+
+user->>browser: select "Submit"
+browser->>runner: POST submit form
+runner->>runner-db: save Submission
+runner->>solidqueue-db: enqueue send submission job
+
+runner->>browser: HTTP 302 response
+browser->>runner: GET confirmation page
+runner->>browser: HTTP 200 response
+browser->>user: Display confirmation page
+
+```
+
+## Sending the submission email asynchronously
+
+```mermaid
+
+---
+title: Sending the submission email asynchronously 
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Forms runner worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant s3 as Amazon S3
+participant ses as Amazon SES
+participant sns as Amazon SNS
+participant inbox as Email inbox
+participant sentry as Sentry
+
+actor processor
+
+worker->>solidqueue-db: dequeue send submission job
+worker->>runner-db: get Submission
+worker->>s3: get file(s)
+worker->>ses: send email
+
+break error 
+  alt AWS SDK error and max retries not reached
+    worker->>solidqueue-db: schedule retry
+  else
+    worker->>sentry: send error
+  end
+end
+
+ses->>worker: return message_id
+worker->>runner-db: set mail_message_id on Submission
+worker->>runner-db: update mail_status of Submission to "pending"
+
+ses-)inbox: send email
+note over ses,inbox: happens some time later
+
+alt email sent successfully
+  ses->>sns: send delivery notification
+  processor->>inbox: get form from inbox
+  processor->>processor: process form
+else email bounces
+  ses->>sns: send bounce notification
+  note over ses,sns: We have an SQS queue subscribed to the<br/> SNS topic and a recurring task to poll<br/>the SQS queue.
+end
+
+worker->>worker: Recurring task deletes Submissions<br/>that haven't bounced after 7 days
+```
+
+## Handling email bounces/complaints
+```mermaid
+
+---
+title: Handling email bounces/complaints
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Solid Queue worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant sqs as Amazon SQS
+participant sentry as Sentry
+
+actor support as Forms team tech support
+
+worker->>solidqueue-db: enqueue recurring receive bounces job
+worker->>solidqueue-db: dequeue receive bounces job
+worker->>sqs: get messages from bounces and complaints queue
+alt there is a bounce SQS message
+  worker->>runner-db: get Submission by the message_id in the SQS message
+  worker->>runner-db: update delivery_status of Submission to "bounced"
+  worker->>worker: Log with the submission details
+  worker->>sentry: send error event
+  sentry->>support: Alert via Slack
+  support->>support: Identify why the email bounced
+  support->>support: Run rake task to retry submission
+else there is a complaint SQS message
+  worker->>runner-db: get Submission by the message_id in the SQS message
+  worker->>worker: Log with the submission details
+end
+
+```
+
+## Handling successful deliveries
+
+```mermaid
+
+---
+title: Handling successful deliveries
+---
+
+sequenceDiagram
+
+autonumber
+
+participant worker as Solid Queue worker
+participant solidqueue-db as Solid Queue database
+participant runner-db as Forms Runner database
+participant sqs as Amazon SQS
+
+actor support as Forms team tech support
+
+worker->>solidqueue-db: enqueue recurring receive deliveries job
+worker->>solidqueue-db: dequeue receive deliveries job
+worker->>sqs: get messages from deliveries queue
+worker->>runner-db: get Submission by the message_id in the SQS message
+worker->>worker: log a "form_submission_delivered" event
+note over worker,runner-db: we don't currently use the "delivered" status for anything other than for information
+```