Two pods, two emails

A post after the series CQRS with effect-cqrs, on its lending library. November 2026.

The library service runs fine on one machine. Then traffic grows, or someone wants zero downtime deploys, and the Kubernetes deployment goes from one replica to two. Commands are fine: both replicas append to the same store, and optimistic concurrency settles any race. The next morning a member writes in: why did I get two emails for one loan?

Everything that reads the log in the background, view models, sagas and outboxes, now runs twice. This post shows what goes wrong and the fix. Every snippet runs in examples/blog/two-pods.test.ts.

Emails through an outbox

The library emails a member when a book is lent to them. An email is something that must happen once, outside the system, so it goes through an outbox, which reads the log from its checkpoint and sends what it finds:

examples/blog/two-pods.ts

export interface Email {
  readonly to: string
  readonly subject: string
}

/** The mail service both replicas send through. It keeps what it sent. */
export class Mailer extends Context.Service<Mailer, {
  send(email: Email): Effect.Effect<void>
  readonly sent: Effect.Effect<ReadonlyArray<Email>>
}>()("library/Mailer") {
  static readonly layer = Layer.effect(
    Mailer,
    Effect.gen(function*() {
      const sent = yield* Ref.make<ReadonlyArray<Email>>([])
      return Mailer.of({ send: (email) => Ref.update(sent, (all) => [...all, email]), sent: Ref.get(sent) })
    })
  )
}

// Members are emailed when a book is lent to them.
export const Notifications = Outbox.make({
  name: "notifications",
  aggregates: [Book],
  messages: {
    BookLent: ({ memberId }, { aggregateId }) => ({
      type: "email",
      payload: { to: memberId, subject: `You borrowed ${aggregateId}` }
    })
  },
  publish: (messages) =>
    Mailer.use((mailer) => Effect.forEach(messages, ({ payload }) => mailer.send(payload as Email), { discard: true }))
})

Each replica runs the outbox, against what they share: the event store, the checkpoints and the mail service.

examples/blog/two-pods.ts

// What each replica runs: the outbox, on the shared store, checkpoints and mailer.
const replica = Layer.fresh(Outbox.layer([Notifications]))

// What the replicas share: in production, the database and the mail service.
export const Shared = Layer.mergeAll(EventStore.layerMemory, Checkpoints.layerMemory, Mailer.layer)

Two replicas

examples/blog/two-pods.ts

const first = yield* Scope.make()
const second = yield* Scope.make()
const replicaA = Context.get(yield* Layer.buildWithScope(replica, first), Outbox.Outboxes)
const replicaB = Context.get(yield* Layer.buildWithScope(replica, second), Outbox.Outboxes)

yield* books.commands.add("book-1", { title: "Dune", author: "Frank Herbert" })
const lent = yield* books.commands.lend("book-1", { loanId: "loan-1", memberId: "alice" })
yield* replicaA.published(lent.position)
yield* replicaB.published(lent.position)

const afterOneLoan = yield* mailer.sent

With nothing else, afterOneLoan holds Alice’s email twice. Each replica read the BookLent, from the same checkpoint, and each sent it. Both did their job. Neither knew the other existed.

It’s not only emails. Every view model in both replicas handles every event:

• Read models in a shared database get every update twice. Setting a value twice is harmless, but adding to a count, appending to a list or inserting a row is not.
• Checkpoints race. Both replicas save their position under the same name, and the slower one can move it back, so events are handled again after a restart.
• Sagas start twice. The trigger is a view model. The workflow engine’s idempotency key catches the duplicate start, if it is durable and shared, but every trigger still runs on both.

Read models kept in each replica’s memory are the exception: each replica should build its own. That’s why running everything everywhere is the default.

One holder, one standby

The fix is to decide, for each view model, which replica runs it. effect-cqrs calls this a lease: ViewModel.layer takes one per view model before running it, from the Leases service in context. The replica holding the lease runs the view model, and the others wait as standbys.

In production that is Leases.layerPostgres(), which holds each lease as a PostgreSQL advisory lock, on the same database as the events:

const Live = Layer.mergeAll(ViewModel.layer([Catalog]), Outbox.layer([Notifications])).pipe(
  Layer.provideMerge(Layer.mergeAll(
    PostgresEventStore.layer(),
    PostgresEventStore.layerCheckpoints(),
    Leases.layerPostgres()
  ))
)

The example runs both replicas in one test, so it uses leases in memory that work the same way: the first to ask holds the lease, and the next one in line gets it when the holder goes away.

examples/blog/two-pods.ts

/**
 * Leases in memory that two replicas in one process can share: the first to ask holds a
 * lease, and when its scope closes the next one in line gets it. `Leases.layerPostgres()`
 * does the same across processes, with PostgreSQL advisory locks.
 */
export const sharedLeases = Layer.sync(Leases.Leases, () => {
  const lines = new Map<string, ReadonlyArray<SubscriptionRef.SubscriptionRef<boolean>>>()
  return Leases.Leases.of({
    acquire: (name) =>
      Effect.gen(function*() {
        const held = yield* SubscriptionRef.make(false)
        const line = lines.get(name) ?? []
        lines.set(name, [...line, held])
        if (line.length === 0) yield* SubscriptionRef.set(held, true)
        yield* Effect.addFinalizer(() =>
          Effect.gen(function*() {
            const rest = (lines.get(name) ?? []).filter((ref) => ref !== held)
            lines.set(name, rest)
            const next = rest[0]
            if (next !== undefined) yield* SubscriptionRef.set(next, true)
          })
        )
        return { held: SubscriptionRef.get(held), changes: SubscriptionRef.changes(held) }
      })
  })
})

With them, the same two replicas send Alice one email. Replica A holds the lease and runs the outbox. Replica B’s outbox waits, and its status says holdsLease: false.

The holder goes away

Replicas don’t live forever. A deploy replaces them, a node is drained, a process runs out of memory. When the holder stops, its lease is released, and the standby takes over from the checkpoint the holder saved:

examples/blog/two-pods.ts

// Replica A is stopped: a deploy, a crash, the node it ran on is drained.
yield* Scope.close(first, Exit.void)

yield* books.commands.add("book-2", { title: "Emma", author: "Jane Austen" })
const lentToBob = yield* books.commands.lend("book-2", { loanId: "loan-2", memberId: "bob" })
yield* holding(replicaB)
yield* replicaB.published(lentToBob.position)

const afterFailover = yield* mailer.sent

Bob gets his email from replica B. Nothing was sent twice and nothing was skipped, because the checkpoint is shared and only ever moved by the holder.

With PostgreSQL, the lock lives on a database connection, so it goes when the process dies or loses its connection. Standbys try to take over every 5 seconds by default.

Caution

A holder notices it lost its connection at its next check, so for up to one interval the old holder and the new one can both handle the same events. A lease makes duplicates rare, not impossible. Consumers of an outbox still deduplicate on the message id, and handlers that write to a shared database still write idempotently.

What else changes with replicas

• caughtUp waits only for local view models. On a standby it doesn’t wait for the view models another replica runs, since nothing in this process moves them. Read your own writes where the view model runs, or from the shared read model directly.
• replay and retryParked run on the holder. A standby refuses them.
• In an Effect cluster, Singleton from effect/cluster does the same job with the cluster’s shard ownership, and needs no leases.

Rules of thumb

• Commands scale out, background work runs once. Any number of replicas can take commands. Each view model, saga trigger and outbox runs in one of them at a time.
• Turn leases on before the second replica. The first duplicate email is the cheapest reminder you will get.
• Read models in memory run everywhere, shared ones run once. Leases are for view models whose data, or effects, are shared.
• Stay idempotent anyway. Failover has a window, and at-least-once delivery has always allowed a repeat.

Next

The view models guide covers leases and standbys in full, and The double write covers outboxes.