--- title: "About Pneumail's architecture" description: "How Pneumail sends 250,000 opt-in emails a day without duplicates, and why it is built this way." sidebar: order: 1 --- Pneumail is built to send 250,000 opt-in emails a day through one Amazon SES account, for several workspaces at once. It must never send one person the same campaign twice, and one workspace's bad list must not hurt another. This page explains the design that does that and why. The README is the user's view and the spec. `adr/0001-delivery-ledger.md` records how the design was chosen. ## Three constraints set the design - SES `SendEmail` has no idempotency token. When a call times out, the response cannot tell us whether SES took the message. - One SES account has one rate limit and one daily quota, shared by every workspace. All tenants' combined sending also affects the account's reputation, so tenant isolation alone does not protect the account. - Gmail does not report individual spam complaints to SES. Gmail is most of a typical consumer list, so complaint rates from SES events miss most complaints. ## Every message is a row in one ledger A campaign, once its send starts, becomes an immutable run with exactly one delivery row per recipient address. The send attempt, the SES message ID, the events, and the stats all attach to that row. A database constraint, `UNIQUE (workspace_id, run_id, address_key)`, makes a second delivery for the same person impossible. ### Where state lives | State | Store | Sole writer | |---|---|---| | Workspaces, members, API keys (hashed), scopes | Postgres | `access` | | Sending domains, SES tenant and configuration set bindings | Postgres | `identity` | | Publications, sender and footer revisions | Postgres | `publications` | | Contacts, consent records, publication memberships | Postgres | `audience` | | Suppression | Postgres | `safety` | | Campaign documents (block JSON), immutable revisions | Postgres JSONB | `campaigns` | | Rendered HTML and images | R2, content-addressed | `composer` | | Runs, delivery ledger, attempts, stats, contact engagement | Postgres | `delivery` (private) | | Send rate, daily quota, domain warmup | One Durable Object per SES account and region | `governor` (private) | | Raw SES and tracking events | R2, compressed daily batches | `archive` (private) | Postgres is PlanetScale Postgres reached through Hyperdrive. Two rules apply to every connection: - Query caching is off. Hyperdrive caches reads for up to 60 seconds by default, and a cached read could send to someone who just unsubscribed. - Every transaction sets the workspace with `SET LOCAL app.workspace_id`, and row-level security reads it. Hyperdrive pools connections per transaction, so a session-level setting would leak between workspaces. Queues carry wakeups and IDs only. A queue never records whether a message was sent. Workflows arrive with automations in chunk 7 and will create deliveries in this same ledger. Delivery rows stay in Postgres for 90 days, about 22 million rows at full volume. Closed runs then move to R2 as compact facts. ## Each workspace is its own SES tenant - One Pneumail workspace is one SES tenant with the Standard reputation policy. SES pauses a tenant that crosses its thresholds and leaves the others sending. - Front Door and Backpack are separate workspaces, so a bad Front Door import cannot pause Backpack's newsletter. - Each workspace has one configuration set per traffic class: `marketing` now, `transactional` later. SES open and click tracking are off, because Pneumail tracks clicks itself. - Each configuration set overrides the account suppression list with SES suppression off. Pneumail is the only owner of suppression, and a test proves that no suppressed address ever reaches the SES adapter. - The configuration set publishes `Send`, `Delivery`, `Bounce`, `Complaint`, `Reject`, `RenderingFailure` and `DeliveryDelay` to EventBridge. EventBridge also delivers tenant status changes. - Because tenants still share the account's fate, Pneumail's own gates (first-batch hold, auto-pause, warmup) protect the account before SES has to. ## Suppression has one owner and one rule table | Event | Scope | Lifetime | |---|---|---| | Hard bounce, SES subtype `NoEmail` or status 5.1.1 or 5.1.10 | The address, in every workspace | 12 months | | Other permanent bounce, except SES subtypes `Suppressed` and `OnAccountSuppressionList` and 5.7.x policy rejections | The workspace | 12 months | | Complaint | The publication | Permanent | | Unsubscribe | The publication | Permanent | | Mailbox full | The address | 72 hours | | DNS failure | The address | 1 hour | | Other final soft bounce | The address | 24 hours | Temporary entries carry `until`, and eligibility compares it with database time, so no cleanup job has to run. Re-importing an unsubscribed address never resubscribes it. No API scope can remove an unsubscribe or a complaint. Only the person can, by signing up again with a new consent record. A size or rendering failure holds the campaign and does not touch suppression. ## Consent is a record Every contact's membership in a publication has a consent record with two separate facts. The `method` is how the person agreed: `form` or `double_opt_in`. The `channel` is how the record reached Pneumail: `signup`, `import` or `api`. The record also keeps the source URL (empty for API consents that have none), the time the person consented, and for imports a SHA-256 fingerprint of the file. An import without a method fails before it writes anything. A row without `consented_at` waits for review. Pneumail never edits an address. Suggested typo fixes wait for a person to accept them. ## How a send works 1. Schedule. The API checks the idempotency key, renders the document, and runs the pre-send checks. The approval request covers a digest of the content, the sender, the footer, the segment definition, the schedule, and the audience count. 2. Approve. An admin approves in the app. The approval page requires an admin session. An MCP client that supports URL elicitation opens that page for the human. A client without it falls back to the approval email plus `campaigns wait`. A key with the `send:auto` scope records its approval against the same digest. Editing any covered input cancels the approval. 3. Start. At the scheduled time, one `INSERT ... SELECT` freezes the audience into the run, ordered by most recent human click, then most recent consent. If the count moved more than 10% from the approved count, the run waits for a new approval. 4. First batch. When a run includes contacts this workspace has never sent to, a random 10% of those contacts (at most 1,000) goes first. The run then holds 30 minutes and checks bounces before it continues. The sample is random, not most engaged first, because it measures the quality of the list. 5. Claim. Workers claim batches of ready deliveries with `FOR UPDATE SKIP LOCKED` and a lease generation. A Cron scan of the due-time index recovers lost queue messages. 6. Pace. The governor grants permits against the SES send rate, the rolling 24-hour quota, and the sending domain's warmup limit. A new domain starts at 2,000 a day, and the limit doubles after each day the domain sends with bounces under 2%. The governor has two lanes: sequences go ahead of campaigns. When a limit is spent, the run shows `waiting for quota` and continues the next day. 7. Authorize. One transaction checks holds, membership, consent, suppression and the lease, then marks the attempt `invoking`. After a pause commits, no new attempt passes this step. 8. Send. The SES adapter makes one signed `SendEmail` call per recipient. Message tags carry the workspace, publication, run and delivery IDs. The HTTP client never retries this call by itself. ### Responses are sorted into sent, not sent, and unknown | SES result | Meaning | What happens | |---|---|---| | 200 with a message ID | Accepted. SES can still drop it later, for example for a virus. | Attempt `accepted` | | `TooManyRequestsException` (429) | Not sent, throttled | Retry after backoff | | Daily quota exceeded | Not sent | Run shows `waiting for quota` | | `MessageRejected` | Not sent. The content is invalid, so every recipient would fail. | Run holds | | `MailFromDomainNotVerifiedException`, `NotFoundException`, `BadRequestException` | Not sent. Configuration is wrong. | Run holds | | `SendingPausedException`, `AccountSuspendedException` | Not sent | Run holds, admins emailed | | Timeout, connection reset, 5xx | Unknown | Attempt `unknown`, never retried | A later `Send` event with our tags turns an unknown attempt into `accepted`. Unknown attempts are counted and shown in stats. EventBridge does not deliver events in order and retries for up to 24 hours, so a missing event cannot prove SES never took a message. Dropping one email in a rare timeout costs less than risking a duplicate. ## How events come back 1. EventBridge forwards SES events to a Worker endpoint, which checks a shared secret, puts the event on a Queue, and returns 200. Clicks and opens arrive at our tracking host and go on the same Queue. 2. The consumer reads the tags, checks them against stored bindings, and quarantines anything that does not match. 3. The consumer reduces a whole batch of up to 100 events in one transaction: delivery facts, then suppression, then rolling bounce and complaint rates, then campaign stats and contact engagement, all from the same fact delta. Facts are idempotent, so duplicates and replays change nothing. Batching keeps a burst of Apple Mail opens from fighting over one stats row. 4. Archival to R2 runs on its own, so an R2 outage never delays an unsubscribe. ### When a send pauses A publication holds when hard bounces pass 2% after 100 sends, when complaints pass 0.1% after 1,000 sends, or when the SES tenant is paused. `domains add` also prints a Google verification TXT record, and Pneumail registers the domain in Google Postmaster Tools (API v2beta). A daily job reads the domain's spam rate. One day above 0.1% emails the admins. A 7-day rolling rate above 0.1%, once the domain has enough Gmail volume for Postmaster to report, holds the domain's publications until an admin looks. These limits sit below Gmail's 0.3% line on purpose. Every hold emails every admin with the limit that tripped. ## Tracking, unsubscribe and headers Clicks go through a redirect at `t.pneumail.com` in the MVP. Bot clicks are removed using timing, user agent and network signals. Opens use a pixel and are marked `automatic` when they come from Apple Mail Privacy Protection or a link scanner. Automatic opens never count as engagement. Every message carries: - `List-Unsubscribe` with an HTTPS URL and `List-Unsubscribe-Post: List-Unsubscribe=One-Click`, both covered by DKIM. - `List-ID: ..pneumail.com`. - `Feedback-ID`, built by SES from the tags `ses:feedback-id-a` (the campaign) and `ses:feedback-id-b` (the publication). SES adds the account and `AmazonSES` as the last two fields. The unsubscribe POST works without a login and writes suppression before it returns. A GET only shows the page, so link scanners cannot unsubscribe anyone. ## Agents use the same contract as the app `packages/contract/openapi.yaml` is the only wire contract. Generation produces request validators, the REST client and the CLI command table. About 15 task-level operations carry an `x-mcp` flag and become MCP tools, with `destructiveHint` and `idempotentHint` set. Those annotations are hints for the client. Approval stays the control. A missing handler fails the build. Every operation that can send mail takes an idempotency key, and reusing a key with different content fails. Sandbox workspaces approve sends automatically and route every message to the SES mailbox simulator, so an agent can run the whole README with no DNS and no human. Simulator sends are billed and count against the account's send rate, so sandboxes use a separate AWS account. CI runs the README against a fake provider with a 20-row CSV. ## Module map ``` apps/web/ TanStack Start: app, composer, approvals apps/service/ Worker: REST, MCP, event intake, unsubscribe, tracking, queue and cron handlers packages/contract/ openapi.yaml and generators packages/cli/ generated commands, browser and device-code login packages/mail/src/ access/ identity/ publications/ audience/ composer/ campaigns/ insights/ public modules private/ delivery ledger and reducer, safety, governor, SES adapter, archive, SQL packages/email-editor/ block editor over the email document type docs/ Blume site for docs.pneumail.com ``` Package exports make `private/` unreachable from outside `packages/mail`. Each operation reads in three files: the adapter in `apps/service`, the public module, and its private implementation. ## Core types ```ts type WorkspaceId = Brand type AddressKey = Brand // canonical address, one versioned policy type Consent = { method: "form" | "double_opt_in" // how the person agreed channel: "signup" | "import" | "api" // how the record arrived sourceUrl: string | null consentedAt: Instant fileSha256: string | null // imports only } type Delivery = { runId: RunId address: AddressKey state: "ready" | "leased" | "invoking" | "accepted" | "unknown" | "skipped" | "failed" facts: DeliveryFacts // delivered, bounced(kind), complained, clicked, unsubscribed } type Suppression = { scope: { kind: "address" } | { kind: "workspace" } | { kind: "publication"; publicationId: PublicationId } address: AddressKey reason: "hard_bounce" | "permanent_bounce" | "complaint" | "unsubscribe" | "mailbox_full" | "dns_failure" | "soft_bounce" until: Instant | null // null is permanent } // Callers state intent. Claims, permits and SES never appear on this interface. interface Campaigns { create(ws: WorkspaceId, input: CampaignDraft, key: IdempotencyKey): Promise schedule(ws: WorkspaceId, id: CampaignId, at: Instant, key: IdempotencyKey): Promise approve(ws: WorkspaceId, request: ApprovalRequestId, by: MemberId): Promise pause(ws: WorkspaceId, id: CampaignId, reason: string): Promise resume(ws: WorkspaceId, id: CampaignId, by: MemberId): Promise stats(ws: WorkspaceId, id: CampaignId): Promise } ``` ## Cost at 250,000 emails a day | Item | Monthly | |---|---| | SES sending, 7.5 million emails at $0.10 per 1,000 | $750 | | SES tenant management, $0.005 per 1,000 | $38 | | Cloudflare Workers paid plan and Queues, about 20 million events | about $35 | | EventBridge API destination | about $3 | | PlanetScale Postgres | priced in chunk 1 | That is about $0.11 per 1,000 emails before Postgres. ## Tradeoffs accepted - We accept a managed Postgres dependency in exchange for one transaction around approval, suppression and sending. - We accept dropping a message after an SES timeout in exchange for never sending a duplicate. - We accept one immutable run per campaign. Sending again means cloning the campaign, so stats stay exact. - We accept a shared tracking host while every workspace is ours, in exchange for one less DNS record. - We accept that a `NoEmail` bounce in one workspace suppresses the address in all of them for 12 months. ## Alternatives we rejected - A Durable Object per campaign and per publication, with no central database. Every workspace-wide lookup becomes a fan-out, and one object per campaign caps throughput near 120 sends a second until it is split. - One queue message as the whole recipient record. The queue delivers at least once and cannot say whether a recipient was sent, so every consumer would rebuild deduplication. - A Workflow per recipient. It adds a second lifecycle beside the ledger and does nothing about SES timeouts. - `SendBulkEmail` with SES templates. Per-recipient unsubscribe tokens and tags get awkward, and the volume does not need it. ## Before the first real send - Front Door residents and advisors opted in to updates earlier, so their consent exists. The import records the original form as the consent source, with `method: form` and each person's consent time. - The SES quota increase to 300,000 a day at 50 a second is requested with AWS. Until it lands, the governor holds sends to the current 50,000 a day at 14 a second. ## Open question - How long do we keep raw events? The proposal is 13 months in R2. ## Next step Chunk 1 deploys the skeleton, the Blume docs site and a sandbox workspace. Chunk 2 starts with the delivery reducer and a crash simulator in CI that proves no failure point causes a second SES call.