WPB-27255: move email sending to background-worker

changelog.d/0-release-notes/wpb-27255-email-background-worker.md

@@ -0,0 +1,48 @@
+Outbound email delivery has moved from **brig** to the **background-worker**,
+and the queue now carries the *composing payload* instead of a finished mail:
+brig no longer renders templates or builds MIME mail. It enqueues every
+outbound email (verification, activation, password-reset, invitation,
+new-client, account-deletion, SAML IdP-change, provider and enterprise-audit
+mail) as a `send_email` job on the `emails` Arbiter queue (a PostgreSQL table
+in the default Arbiter schema), carrying only the email type, locale and
+structured inputs (recipient, keys/codes, team names, certificate summaries,
+...). The background-worker composes the email — locale template selection,
+placeholder rendering, MIME building — from the localized templates bundled in
+its image (`/usr/share/wire/templates`) right before performing the actual
+SMTP/SES send. The queue is not routed through RabbitMQ.
+
+Operators must configure two blocks on the background-worker:
+
+- `background-worker.config.email` — the transport (SES **or** SMTP, the same
+  shape as brig's former `emailSMS.email`); for SES also the worker's AWS
+  region and credentials (`AWS_REGION` and
+  `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`).
+- `background-worker.config.emailTemplates` — the template directory, default
+  locale, sender address, branding and the user/team/provider URL templates.
+  These were previously brig's `emailSMS` template/URL/branding settings
+  (`emailSMS.general.templateDir`, `emailSMS.general.templateBranding`,
+  `emailSMS.user.{activation,passwordReset,deletion}Url` and
+  `emailSMS.provider`); those brig keys are gone, and the worker values must
+  match what brig used to configure so emails render with the same URLs and
+  branding as before. The templates directory now ships in the
+  background-worker image.
+
+Failed sends are retried by Arbiter with bounded exponential backoff and
+eventually land in the queue's dead-letter table, so transient
+background-worker downtime does not lose mail: jobs stay in the `emails` table
+until a worker picks them up.
+
+When rolling out, deploy the updated background-worker before (or alongside)
+the updated brig so that the new `send_email` jobs are consumed as soon as they
+appear; both services run the Arbiter migrations that create the `emails`
+table at startup. This ordering assumes no intermediate build that queued email
+on RabbitMQ is still running: `send-email` messages on the `background-jobs`
+queue are requeued forever by an updated worker (which no longer understands
+them). If such a build ran anywhere, drain or delete residual `send-email`
+messages from the `background-jobs` queue before upgrading the worker.
+
+Note: the `emails` queue and its dead-letter table live in the shared
+PostgreSQL database and contain the queued request data (including one-time
+codes, recipient addresses and reset URLs, for jobs that were never delivered).
+Access to the database should therefore be least-privileged, and DLQ growth
+should be monitored.

charts/wire-server/templates/background-worker/configmap.yaml

@@ -80,6 +80,29 @@ data:
       {{- end }}
     {{- end }}
 
+    email:
+    {{- if .useSES }}
+      sesQueue: {{ required "Missing value: background-worker.config.aws.sesQueue" .aws.sesQueue }}
+      sesEndpoint: {{ .aws.sesEndpoint | quote }}
+    {{- else }}
+      smtpEndpoint:
+        host: {{ .smtp.host }}
+        port: {{ .smtp.port }}
+      smtpConnType: {{ .smtp.connType }}
+      {{- if .smtp.username }}
+      smtpCredentials:
+        smtpUsername: {{ .smtp.username }}
+        smtpPassword: {{ .smtp.passwordFile }}
+      {{- end }}
+    {{- end }}
+
+    # Email templates used to compose the emails queued by brig.
+    # These values must match what brig used to configure.
+    {{- with .emailTemplates }}
+    emailTemplates:
+{{ toYaml . | indent 6 }}
+    {{- end }}
+
     migrateConversations: {{ .migrateConversations }}
     migrateConversationCodes: {{ .migrateConversationCodes }}
     migrateTeamFeatures: {{ .migrateTeamFeatures }}

charts/wire-server/templates/background-worker/deployment.yaml

@@ -109,6 +109,20 @@ spec:
           {{ toYaml .Values.additionalVolumeMounts | nindent 10 }}
           {{- end }}
           env:
+          {{- if hasKey $backgroundWorker.secrets "awsKeyId" }}
+          - name: AWS_ACCESS_KEY_ID
+            valueFrom:
+              secretKeyRef:
+                name: background-worker
+                key: awsKeyId
+          - name: AWS_SECRET_ACCESS_KEY
+            valueFrom:
+              secretKeyRef:
+                name: background-worker
+                key: awsSecretKey
+          {{- end }}
+          - name: AWS_REGION
+            value: "{{ $backgroundWorker.config.aws.region }}"
           - name: RABBITMQ_USERNAME
             valueFrom:
               secretKeyRef:

charts/wire-server/templates/background-worker/secret.yaml

@@ -14,6 +14,13 @@ data:
   for_helm_linting: {{ required "No .secrets found in configuration. Did you forget to helm <command> -f path/to/secrets.yaml ?" $backgroundWorker.secrets | quote | b64enc | quote }}
 
   {{- with $backgroundWorker.secrets }}
+  {{- if .awsKeyId }}
+  awsKeyId: {{ .awsKeyId | b64enc | quote }}
+  awsSecretKey: {{ .awsSecretKey | b64enc | quote }}
+  {{- end }}
+  {{- if (not $backgroundWorker.config.useSES) }}
+  smtp-password.txt: {{ .smtpPassword | b64enc | quote }}
+  {{- end }}
   rabbitmqUsername: {{ .rabbitmq.username | b64enc | quote }}
   rabbitmqPassword: {{ .rabbitmq.password | b64enc | quote }}
   {{- end }}

charts/wire-server/templates/brig/configmap.yaml

@@ -144,53 +144,14 @@ data:
         {{- end }}
       {{- end }}
       general:
-        templateDir: /usr/share/wire/templates
         emailSender: {{ .emailSMS.general.emailSender }}
         smsSender: {{ .emailSMS.general.smsSender | quote }}
-        templateBranding:
-          {{- with .emailSMS.general.templateBranding }}
-          brand: {{ .brand }}
-          brandUrl: {{ .brandUrl }}
-          brandLabelUrl: {{ .brandLabelUrl }}
-          brandLogoUrl: {{ .brandLogoUrl }}
-          brandService: {{ .brandService }}
-          copyright: {{ .copyright }}
-          misuse: {{ .misuse }}
-          legal: {{ .legal }}
-          forgot: {{ .forgot }}
-          support: {{ .support }}
-          {{- end }}
 
       user:
       {{- if .emailSMS.user }}
-        activationUrl: {{ .emailSMS.user.activationUrl }}
         smsActivationUrl: {{ .emailSMS.user.smsActivationUrl }}
-        passwordResetUrl: {{ .emailSMS.user.passwordResetUrl }}
-        {{- if .emailSMS.user.invitationUrl }}
-        invitationUrl: {{ .emailSMS.user.invitationUrl }}
-        {{- end }}
-        deletionUrl: {{ .emailSMS.user.deletionUrl }}
       {{- else }}
-        activationUrl: {{ .externalUrls.nginz }}/activate?key=${key}&code=${code}
         smsActivationUrl: {{ .externalUrls.nginz }}/v/${code}
-        passwordResetUrl: {{ .externalUrls.nginz }}/password-reset/${key}?code=${code}
-        invitationUrl: {{ .externalUrls.nginz }}/register?invitation_code=${code}
-        deletionUrl: {{ .externalUrls.nginz }}/users/delete?key=${key}&code=${code}
-      {{- end }}
-
-      provider:
-      {{- if .emailSMS.provider }}
-        homeUrl: {{ .emailSMS.provider.homeUrl }}
-        providerActivationUrl: {{ .emailSMS.provider.providerActivationUrl }}
-        approvalUrl: {{ .emailSMS.provider.approvalUrl }}
-        approvalTo: {{ .emailSMS.provider.approvalTo }}
-        providerPwResetUrl: {{ .emailSMS.provider.providerPwResetUrl }}
-      {{- else }}
-        homeUrl: https://provider.localhost/
-        providerActivationUrl: {{ .externalUrls.nginz }}/provider/activate?key=${key}&code=${code}
-        approvalUrl: {{ .externalUrls.nginz }}/provider/approve?key=${key}&code=${code}
-        approvalTo: success@simulator.amazonses.com
-        providerPwResetUrl: {{ .externalUrls.nginz }}/provider/password-reset?key=\${key}\&code=\${code}
       {{- end }}
 
       team:

charts/wire-server/values.yaml

@@ -372,8 +372,7 @@ galley:
     seccompProfile:
       type: RuntimeDefault
   tests:
-    config:
-      {}
+    config: {}
       # uploadXml:
       #   baseUrl: s3://bucket/path/
 
@@ -1075,6 +1074,66 @@ background-worker:
       # Cron schedule for the cleanup job (0 * * * * = every hour)
       schedule: "0 * * * *"
 
+    # Email transport for the background-worker (delivers the email jobs
+    # enqueued by brig). Same SES/SMTP shape as brig's emailSMS.email.
+    # `useSES` selects the transport: when true, the worker sends via AWS SES
+    # using `aws.sesQueue`/`aws.sesEndpoint` together with the AWS_ACCESS_KEY_ID
+    # / AWS_SECRET_ACCESS_KEY secrets and the AWS_REGION below. When false, it
+    # sends via SMTP using the `smtp.*` settings.
+    useSES: true
+    aws:
+      region: "eu-west-1"
+      sesEndpoint: https://email.eu-west-1.amazonaws.com
+      # sesQueue is required when useSES is true (deployment-specific), e.g.:
+      # sesQueue: wire-brig-events
+
+    # SMTP transport (used when useSES is false). The ConfigMap renders these
+    # into the `email` block; mirrors brig's `smtp` settings.
+    smtp:
+      passwordFile: /etc/wire/background-worker/secrets/smtp-password.txt
+
+    # Email templates used to compose the emails queued by brig.
+    # The background-worker composes all outbound email from these localized
+    # templates (bundled in the image at templateDir); brig no longer has any
+    # template/URL/branding settings, so these values must match what brig
+    # used to configure.
+    emailTemplates:
+      templateDir: /usr/share/wire/templates
+      emailSender: backend@wire.com
+      templateBranding:
+        brand: Wire
+        brandUrl: https://wire.com
+        brandLabelUrl: wire.com
+        brandLogoUrl: https://wire.com/p/img/email/logo-email-black.png
+        brandService: Wire Service Provider
+        copyright: © WIRE SWISS GmbH
+        misuse: misuse@wire.com
+        legal: https://wire.com/legal/
+        forgot: https://wire.com/forgot/
+        support: https://support.wire.com/
+      user:
+        activationUrl: https://<nginz>/activate?key=${key}&code=${code}
+        teamActivationUrl: https://<nginz>/register?team=${team}&team_code=${code}
+        passwordResetUrl: https://<nginz>/password-reset/${key}?code=${code}
+        deletionUrl: https://<nginz>/users/delete?key=${key}&code=${code}
+      team:
+        tInvitationUrl: https://<nginz>/register?team=${team}&team_code=${code}
+        tExistingUserInvitationUrl: https://<nginz>/register?invitation_code=${code}
+        tActivationUrl: https://<nginz>/register?team=${team}&team_code=${code}
+        tCreatorWelcomeUrl: https://example.com/login
+        tMemberWelcomeUrl: https://example.com/download
+      provider:
+        homeUrl: https://provider.localhost/
+        providerActivationUrl: https://<nginz>/provider/activate?key=${key}&code=${code}
+        approvalUrl: https://<nginz>/provider/approve?key=${key}&code=${code}
+        approvalTo: success@simulator.amazonses.com
+        providerPwResetUrl: https://<nginz>/provider/password-reset?key=${key}&code=${code}
+
+  # Optional secret keys (see templates/background-worker/secret.yaml):
+  #   awsKeyId: <aws-access-key-id>          # required for SES; rendered as AWS_ACCESS_KEY_ID
+  #   awsSecretKey: <aws-secret-access-key>   # required for SES; rendered as AWS_SECRET_ACCESS_KEY
+  #   smtpPassword: <smtp-password>           # mounted at /etc/wire/background-worker/secrets/smtp-password.txt;
+  #                                          # consumed via config.smtp.passwordFile (mirrors brig/galley)
   secrets: {}
 
   podSecurityContext:
@@ -1192,19 +1251,6 @@ brig:
       acquisitionTimeout: 10s
       idlenessTimeout: 10m
 
-    emailSMS:
-      general:
-        templateBranding:
-          brand: Wire
-          brandUrl: https://wire.com
-          brandLabelUrl: wire.com
-          brandLogoUrl: https://wire.com/p/img/email/logo-email-black.png
-          brandService: Wire Service Provider
-          copyright: © WIRE SWISS GmbH
-          misuse: misuse@wire.com
-          legal: https://wire.com/legal/
-          forgot: https://wire.com/forgot/
-          support: https://support.wire.com/
     authSettings:
       keyIndex: 1
       userTokenTimeout: 4838400
@@ -1324,8 +1370,7 @@ brig:
     seccompProfile:
       type: RuntimeDefault
   tests:
-    config:
-      {}
+    config: {}
       # uploadXml:
       #   baseUrl: s3://bucket/path/
 

docs/src/developer/reference/config-options.md

@@ -2448,3 +2448,118 @@ Notes
 - `jobs.workerThreads` controls the number of worker threads in each job queue. The default is `1`; increasing it allows jobs in that queue to run in parallel when their group keys permit it.
 - Both job queues share the same PostgreSQL pool. Increasing `jobs.workerThreads` can increase the number of connections needed when more jobs run concurrently, but it does not create a permanently dedicated connection per thread or queue.
 - The job runner is poll-only and does not require an additional PostgreSQL listener connection.
+
+## Background worker: Email sending
+
+
+The background-worker delivers the email jobs enqueued by brig on the `emails`
+Arbiter queue (PostgreSQL). It requires an `email` transport (AWS SES or SMTP),
+the same shape brig uses for `emailSMS.email`. Configuration is supplied via
+Helm under `background-worker.config` and rendered into the `email` block of
+`background-worker.yaml`.
+
+The transport is selected by `background-worker.config.useSES`:
+
+- `useSES: true` (default) renders an SES block. `aws.sesQueue` is required and
+  `aws.sesEndpoint` selects the SES endpoint. The worker also needs the
+  `AWS_REGION`, `AWS_ACCESS_KEY_ID`, and `AWS_SECRET_ACCESS_KEY` environment
+  variables, injected from `background-worker.config.aws.region` and the
+  `awsKeyId`/`awsSecretKey` secrets (the same pattern brig uses).
+- `useSES: false` renders an SMTP block using the `smtp.*` settings. The SMTP
+  password is read from the file named by `smtp.passwordFile` (mounted from the
+  `smtpPassword` secret).
+
+Rendered config (`background-worker.yaml`):
+
+```yaml
+# SES:
+email:
+  sesQueue: wire-brig-events
+  sesEndpoint: https://email.eu-west-1.amazonaws.com
+# SMTP (xor SES):
+# email:
+#   smtpEndpoint: { host: smtp.example.com, port: 587 }
+#   smtpConnType: tls
+#   smtpCredentials:
+#     smtpUsername: wire
+#     smtpPassword: /etc/wire/background-worker/secrets/smtp-password.txt
+```
+
+Helm values (under `background-worker`):
+
+```yaml
+config:
+  useSES: true
+  aws:
+    region: "eu-west-1"
+    sesEndpoint: https://email.eu-west-1.amazonaws.com
+    sesQueue: wire-brig-events   # required when useSES is true
+  smtp:
+    passwordFile: /etc/wire/background-worker/secrets/smtp-password.txt
+secrets:
+  awsKeyId: <aws-access-key-id>          # SES only
+  awsSecretKey: <aws-secret-access-key>  # SES only
+  smtpPassword: <smtp-password>          # SMTP only
+```
+
+Notes
+
+- `email` is required: the worker fails to start without a transport.
+- For SES, the worker reads `AWS_REGION` from `config.aws.region` and the AWS
+  credentials from the `awsKeyId`/`awsSecretKey` secrets, mirroring brig.
+- For SMTP, the password is mounted at
+  `/etc/wire/background-worker/secrets/smtp-password.txt` (from the
+  `smtpPassword` secret); `config.smtp.passwordFile` must point at it.
+- Email jobs are inserted by brig into the `emails` table of the default
+  Arbiter schema (created at startup by the Arbiter migrations). Failed sends
+  are retried with bounded exponential backoff and eventually moved to the
+  queue's dead-letter queue, so transient worker downtime does not lose email
+  jobs.
+- The `emails` queue and its dead-letter table live in the shared PostgreSQL
+  database and contain the queued request data (including one-time codes,
+  recipient addresses and reset URLs for jobs that were never delivered). Keep
+  database access least-privileged and monitor DLQ growth.
+
+## Background worker: Email templates
+
+The background-worker does not only deliver email, it **composes** it: brig
+enqueues the composing payload (email type, locale and inputs such as
+recipient, keys/codes, team names) as `send_email` jobs, and the worker
+selects the localized template, renders the placeholders and builds the MIME
+mail right before sending. The templates directory ships in the
+background-worker image at `/usr/share/wire/templates`.
+
+Configure it via Helm under `background-worker.config.emailTemplates`
+(rendered into the `emailTemplates` block of `background-worker.yaml`). These
+settings were previously brig's `emailSMS` template/URL/branding settings;
+brig no longer has them, so deployments must carry them on the worker instead
+(they must match what brig used to configure, or emails will render with
+different URLs/branding than before):
+
+```yaml
+config:
+  emailTemplates:
+    templateDir: /usr/share/wire/templates
+    # defaultLocale: en       # optional; falls back to en
+    emailSender: backend@wire.com
+    templateBranding:         # the 10 branding placeholders
+      brand: Wire
+      brandUrl: https://wire.com
+      # ...
+    user:                     # user email URL templates
+      activationUrl: https://<nginz>/activate?key=${key}&code=${code}
+      teamActivationUrl: https://<nginz>/register?team=${team}&team_code=${code}
+      passwordResetUrl: https://<nginz>/password-reset/${key}?code=${code}
+      deletionUrl: https://<nginz>/users/delete?key=${key}&code=${code}
+    team:                     # team email URL templates
+      tInvitationUrl: https://<nginz>/register?team=${team}&team_code=${code}
+      # ...
+    provider:                 # provider email URL templates
+      homeUrl: https://provider.example.com/
+      # ...
+```
+
+brig still configures `emailSMS.general.emailSender` (used for SCIM
+invitations and the enterprise audit email configuration) and the team
+invitation URL templates (`emailSMS.team`, rendered into API responses);
+everything else email-related lives on the worker.