The double write

A post after the series CQRS with effect-cqrs, on its lending library. November 2026.

Sooner or later the library has to tell someone else. The notifications service emails members their due dates, the statistics team counts loans, the city’s network of libraries wants to know which books are out. They live in other processes, and they hear about loans through a message broker: Kafka, NATS, SQS, RabbitMQ.

So a loan is two writes: one to our database, one to the broker. This post is about the moment between them, which is where a whole class of production bugs lives. Every snippet runs in examples/blog/double-write.test.ts.

A broker that is sometimes down

The example’s broker keeps messages in memory and can be told to fail the next few sends, the way a real one does when the network drops or it is restarting:

examples/blog/double-write.ts

export class BrokerDown extends Schema.TaggedError<BrokerDown>()("BrokerDown", {}) {}

export interface Sent {
  readonly id: string | null
  readonly type: string
  readonly payload: unknown
}

/** A message broker in memory, which can be told to fail the next few sends. */
export class Broker extends Context.Service<Broker, {
  send(messages: ReadonlyArray<Sent>): Effect.Effect<void, BrokerDown>
  readonly received: Effect.Effect<ReadonlyArray<Sent>>
  failNext(times: number): Effect.Effect<void>
}>()("library/Broker") {
  static readonly layer = Layer.effect(
    Broker,
    Effect.gen(function*() {
      const received = yield* Ref.make<ReadonlyArray<Sent>>([])
      const failures = yield* Ref.make(0)
      return Broker.of({
        send: (messages) =>
          Ref.modify(failures, (left) => [left > 0, Math.max(0, left - 1)]).pipe(
            Effect.flatMap((down) =>
              down ? Effect.fail(new BrokerDown()) : Ref.update(received, (all) => [...all, ...messages])
            )
          ),
        received: Ref.get(received),
        failNext: (times) => Ref.set(failures, times)
      })
    })
  )
}

Store, then send

The obvious version stores the loan, then publishes it:

examples/blog/double-write.ts

export const storeThenSend = Effect.fnUntraced(function*(bookId: string, loanId: string, memberId: string) {
  const books = yield* Books
  const broker = yield* Broker
  yield* books.commands.lend(bookId, { loanId, memberId })
  yield* broker.send([{ id: null, type: "library.book-lent.v1", payload: { bookId, loanId, memberId } }])
})

On a day the broker hiccups for one send:

examples/blog/double-write.ts

yield* books.commands.add("book-1", { title: "Dune", author: "Frank Herbert" })
yield* broker.failNext(1)

const failed = yield* Effect.flip(storeThenSend("book-1", "loan-1", "alice"))
// failed._tag: "BrokerDown", so the front desk tells Alice it didn't work

const book = yield* books.state("book-1")
// { status: "lent", loan: { loanId: "loan-1", memberId: "alice", … }, … }
const afterStoring = yield* broker.received
// []: nobody else will ever hear of loan-1

The loan is stored. The message is lost. And the request failed, so the front desk tells Alice that the loan didn’t work while the library’s own records say she has the book. If the desk sends the same request again, lend records nothing for a loan it already has, and the send gets through this time. If nobody does, because Alice walked off with the book or tried another one, the notifications service never emails her a due date, and the statistics are one loan short, forever.

A crash between the two lines does the same: a deploy, an out-of-memory kill, a pod moved to another node. There is no line of code that can run “after the commit, guaranteed”. A retry around the send helps with a short hiccup and does nothing for a process that is gone.

Send, then store

So swap them. If the message must not be lost, send it first:

examples/blog/double-write.ts

export const sendThenStore = Effect.fnUntraced(function*(bookId: string, loanId: string, memberId: string) {
  const books = yield* Books
  const broker = yield* Broker
  yield* broker.send([{ id: null, type: "library.book-lent.v1", payload: { bookId, loanId, memberId } }])
  yield* books.commands.lend(bookId, { loanId, memberId })
})

Now the failure goes the other way:

examples/blog/double-write.ts

const refused = yield* Effect.flip(sendThenStore("book-1", "loan-2", "bob"))
// refused._tag: "BookAlreadyLent"

const afterSending = yield* broker.received
// [{ type: "library.book-lent.v1", payload: { bookId: "book-1", loanId: "loan-2", memberId: "bob" } }]:
// everybody heard of a loan that never happened

The broker accepted the message. Then the book refused, because Alice has it. Every consumer now believes Bob borrowed Dune: Bob gets an email with a due date for a book he doesn’t have, and the statistics count a loan that never happened. The command could also fail for duller reasons, a lost connection or a version conflict, with the same result.

Sending first means announcing a decision before it is made, and the aggregate is the one that makes it.

Note

What about a transaction around both? A database transaction covers the database. The broker is another system, with its own idea of committed, and the protocols that span the two, such as two-phase commit, are slow, fragile and mostly unsupported by brokers. In practice there is no transaction to put them in.

The outbox

The usual fix in a CRUD application is the transactional outbox: in the same transaction as the change, insert a row into an outbox table, and let a separate process read that table and publish what it finds, marking rows as sent. One transaction, one database, and the broker is only ever written from a table that already committed.

With event sourcing you already have that table. The events are written in one append, they are never changed, and they have a position. A process that reads the log from where it last stopped and publishes what it finds is an outbox, without the extra table and without anyone having to remember to insert into it. effect-cqrs calls it Outbox:

examples/blog/double-write.ts

export const LoansOutbox = Outbox.make({
  name: "loans-outbox",
  aggregates: [Book],
  messages: {
    BookLent: ({ loanId, memberId }, { aggregateId }) => ({
      type: "library.book-lent.v1",
      payload: { bookId: aggregateId, loanId, memberId }
    }),
    BookReturned: ({ late, loanId }, { aggregateId }) => ({
      type: "library.book-returned.v1",
      payload: { bookId: aggregateId, loanId, late }
    })
  },
  publish: (messages) =>
    Broker.use((broker) => broker.send(messages.map(({ id, payload, type }) => ({ id, type, payload }))))
})

export const Library = Outbox.layer([LoansOutbox]).pipe(
  Layer.provideMerge(Books.layer),
  Layer.provideMerge(Layer.mergeAll(EventStore.layerMemory, Checkpoints.layerMemory, Broker.layer))
)

messages turns domain events into integration events: a type with a version in it, and a payload that is a contract with other teams, which is not the same as the event’s shape. Events without a mapping, like BookAdded, are not published.

The same bad day, with the outbox:

examples/blog/double-write.ts

yield* books.commands.add("book-1", { title: "Dune", author: "Frank Herbert" })
yield* broker.failNext(2)

// The command only stores. The broker isn't on its path, so it can't fail it.
const { position } = yield* books.commands.lend("book-1", { loanId: "loan-1", memberId: "alice" })

// The outbox tries, fails, waits a little longer each time, and gets through.
// `published` fails while the broker is down, so the example asks again.
yield* outboxes.published(position).pipe(Effect.retry(Schedule.spaced("50 millis")))
const received = yield* broker.received
// [{ id: "book/book-1@2", type: "library.book-lent.v1", payload: { bookId: "book-1", loanId: "loan-1", … } }]

The command succeeded although the broker was down, because the broker was never on its path: a command only appends. The outbox runs in the background, like a view model. It read the BookLent, failed to publish it, waited, tried again with growing pauses, and got through when the broker came back. Alice was told the truth at the desk, and everybody else heard about her loan a little later.

What it costs

• Delivery is at least once. The outbox saves its checkpoint after a batch is published. If the process dies in between, the batch is published again after the restart. Each message has an id, the event’s id, such as book/book-1@2, which never changes, so consumers keep the ids they have handled and skip repeats. Exactly once is not on offer from any broker either: at least once plus deduplication is how it’s done.
• The others hear a little later. Usually milliseconds: a batch goes out when it is full or after linger, ten milliseconds by default. While the broker is down, it is longer, and status shows the outbox retrying.
• Order is per log. Messages leave in the order the events were stored, and a failed batch is retried before anything after it. A consumer that reads in parallel has to keep order per aggregate itself, as Kafka does with partition keys.
• The checkpoint must be durable. With checkpoints in memory, an outbox publishes the whole log again after every restart. Keep them in the database, next to the events.

Rules of thumb

• Never write to two systems in one request. One of the writes has to come from the other, after it has committed.
• A command only stores. Whatever has to happen because of a decision reads it from the log: read models, sagas and outboxes.
• Publish integration events, not your events. Map them in the outbox, give them a versioned type, and leave out what other teams shouldn’t depend on, or keep: personal data in particular.
• Consumers deduplicate on the message id. It is the same every time a message is published.

Next

The integration guide covers Outbox in full, and The saga you skipped shows the same two-writes problem between two aggregates, where a saga plays the outbox’s part.