Operations

Health checks#

/health deliberately has no dependencies. If it checked Postgres, a database blip would make the orchestrator kill an API that was about to recover.

/ready distinguishes two failure modes:

{ "status": "degraded", "checks": {
    "database": { "ok": true },
    "redis":    { "ok": true },
    "mail":     { "ok": false, "error": "connect ECONNREFUSED" } } }

degraded returns 200: the MTA being down means mail queues up rather than being lost, and the API should stay in the load balancer to keep accepting it. Only a database or Redis failure returns 503.

---

Metrics#

Prometheus exposition at GET /metrics, gated behind X-Internal-Token and blocked at nginx — queue depths and volumes are infrastructure detail.

curl -s http://localhost:4000/metrics -H "X-Internal-Token: $INTERNAL_API_TOKEN"

mail_queue_depth{queue="email-send"} 3
mail_queue_depth{queue="email-send:failed"} 0
mail_queue_depth{queue="bounce-processing"} 0
mail_queue_depth{queue="webhook-delivery"} 1
mail_emails_last_hour{status="delivered"} 412
mail_emails_last_hour{status="bounced"} 7
mail_webhook_failures_last_hour 0
mail_api_uptime_seconds 84213

Scrape config:

scrape_configs:
  - job_name: mail-platform
    metrics_path: /metrics
    static_configs: [{ targets: ["api:4000"] }]
    authorization: { type: Bearer, credentials: "" }
    params: {}
    # The token goes in a header:
    http_headers:
      X-Internal-Token: { values: ["<INTERNAL_API_TOKEN>"] }

Alerts worth having#

---

Structured logs#

Every log line is JSON with a stable event field, so they are greppable without parsing prose.

{"level":"info","time":"2026-08-16T12:00:03.912Z","service":"worker","event":"email_delivered","emailId":"email_9f3k…","recipient":"user@gmail.com","dsn":"2.0.0","msg":"delivery status from maillog"}

Secrets are redacted at the logger, not the call site: authorization, cookie, password, apiKey, token, privateKey, dkimPrivateKeyEncrypted and secret are replaced with [redacted] wherever they appear.

docker compose logs -f worker | jq 'select(.event | startswith("email_"))'
docker compose logs api | jq 'select(.level == "error")'

---

Abuse response#

GET /v1/stats/health-signals evaluates one account. The same evaluation runs automatically after sends, at most once a minute per account, and writes an abuse.signals_detected audit entry.

ABUSE_AUTO_SUSPEND=false by default — signals are recorded and logged but a human decides. Turn it on once the thresholds are calibrated to your traffic.

Suspending an account#

docker compose exec postgres psql -U mail -d mail -c \
  "UPDATE users SET status='SUSPENDED', \"suspendedAt\"=now(), \"suspensionReason\"='Spam complaints' WHERE email='abuser@example.com';"

Effective immediately: every API key and session for that account starts returning ACCOUNT_SUSPENDED. Queued messages still in flight are stopped at the next check because the send processor re-validates before handing anything to the MTA.

Reinstating:

docker compose exec postgres psql -U mail -d mail -c \
  "UPDATE users SET status='ACTIVE', \"suspendedAt\"=NULL, \"suspensionReason\"=NULL WHERE email='abuser@example.com';"

Adjusting one account's limits#

docker compose exec postgres psql -U mail -d mail -c \
  "UPDATE users SET \"emailsPerHour\"=500, \"emailsPerDay\"=5000 WHERE email='trusted@example.com';"

NULL means "use the platform default".

---

Audit log#

Every state change is recorded in audit_logs with the actor, IP and user agent.

SELECT "createdAt", action, resource, "resourceId", ip
FROM audit_logs
WHERE "userId" = 'usr_…'
ORDER BY "createdAt" DESC
LIMIT 50;

Recorded actions include user.registered, user.login, user.login_failed, user.password_changed, user.suspended, domain.created, domain.verification_attempted, domain.deleted, api_key.created, api_key.revoked, webhook.created, suppression.created, suppression.deleted, settings.limits_updated, abuse.signals_detected.

Repeated user.login_failed from one IP is the signal to look for.

---

Runbook#

Messages stuck in queued#

docker compose ps worker
docker compose logs --tail=100 worker
curl -s http://localhost:4000/metrics -H "X-Internal-Token: $INTERNAL_API_TOKEN" | grep queue_depth

Usually the worker is down or Redis is unreachable. docker compose up -d worker.

Messages reach sent but never delivered#

sent means Postfix accepted it. delivered comes from parsing the Postfix log, so this pattern almost always means the log tailer is not reading.

docker compose logs worker | grep maillog
docker compose exec worker ls -la /var/log/postfix/mail.log
docker compose exec postfix tail -20 /var/log/postfix/mail.log

If you see maillog_missing, the postfix_logs volume is not mounted into the worker. Check docker-compose.yml.

Bounces are not being recorded#

# Is mail for the bounce host reaching us at all?
docker compose logs postfix | grep bounce-handler

# Can the pipe script reach the API?
docker compose exec postfix sh -c 'curl -sS -o /dev/null -w "%{http_code}\n" \
  -X POST -H "X-Internal-Token: $INTERNAL_API_TOKEN" \
  --data-binary "test" http://api:4000/internal/bounces'

Then confirm bounce.yourservice.com has an MX pointing at the server and that inbound port 25 is open. bounce_unmatched in the worker log means reports are arriving but cannot be tied to a message — check that BOUNCE_HOST matches what the DNS says.

A verified domain stopped working#

docker compose logs worker | grep dns_verification_regressed
dig +short TXT selector1._domainkey.<customer domain>

The periodic sweep demotes a domain whose required records disappear. The customer must republish; then hit Verify DNS again.

Redis was flushed#

Queued jobs are gone; the email rows remain in queued. Requeue them:

SELECT id FROM emails WHERE status = 'queued' AND "createdAt" > now() - interval '1 day';

Re-POST those, or add jobs directly with the email id as the job id. Rate-limit counters rebuild themselves on the next request.

Disk filling up#

The Postfix spool and the maillog grow. Rotate the log:

docker compose exec postfix sh -c 'cp /var/log/postfix/mail.log /var/log/postfix/mail.log.1 && : > /var/log/postfix/mail.log'

The tailer detects the truncation (file smaller than its offset), resets to zero and carries on without replaying or skipping.

Old email bodies are the other growth source. Trim them once you no longer need the content:

UPDATE emails SET html = NULL, text = NULL
WHERE "createdAt" < now() - interval '90 days' AND html IS NOT NULL;

Statuses, events and recipients are small; keep them.