Personal data

The log is append-only, so an event can’t be edited or deleted when someone asks to be forgotten, as the GDPR allows. Crypto-shredding gets around that: the personal fields of an event are encrypted with a key that belongs to the person they describe, the subject, and forgetting the subject destroys the key. The event stays where it is with its amounts, dates and ids, so streams still replay and balances still add up, while its personal fields read as a replacement.

Marking fields

PersonalData.field marks a field of an event and says what it reads as once its subject is forgotten:

import { PersonalData } from "effect-cqrs"

const AccountEvent = Schema.TaggedUnion({
  AccountOpened: {
    owner: PersonalData.field(Schema.String, { redacted: "[forgotten]" }),
    email: Schema.optional(PersonalData.field(Schema.String, { redacted: "forgotten@example.invalid" })),
    currency: Schema.Literals(["CHF", "EUR"])
  },
  TransferSent: {
    to: Schema.String,
    beneficiary: PersonalData.field(Schema.String, { redacted: "[forgotten]", subject: "to" }),
    amount: Schema.Number
  }
})

• The subject is the aggregate id of the stream, unless subject names a property next to the field. Above, beneficiary belongs to the account in to, so forgetting the receiver of a transfer leaves the sender’s events readable.
• The replacement has to be a valid value of the field. It is checked when the field is defined.
• Where a field can be: a property of the event, of a nested struct, optional or nullable. Building an aggregate fails when a personal field sits in an array, a record or an untagged union, is transformed after being marked, or names a subject that isn’t next to it or isn’t always a string. Mark the transformed schema instead: PersonalData.field(Schema.String.pipe(Schema.decodeTo(...)), ...).

Schemas stay plain: a personal field is an ordinary string to projections, handlers and tests. Encryption happens where AggregateService writes to and reads from the store, so nothing else can write a personal field unencrypted.

Keys

An aggregate with personal fields needs PersonalData.Keys, and building its service fails without it. Each subject gets its own AES-256 key the first time it is written. The key store only holds that key wrapped by a MasterKey:

const Keys = PersonalData.layerSql().pipe(
  Layer.provide(PersonalData.layerMasterKeyConfig()),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)

Key store	For
PersonalData.layerMemory()	Tests and the browser. Keys are gone after a restart, and with them everything they encrypted.
PersonalData.layerSql({ table })	SQLite or PostgreSQL, through any effect/sql client

Master key	For
layerMasterKeyConfig()	32 bytes in base64 from PERSONAL_DATA_MASTER_KEY, with its id in PERSONAL_DATA_MASTER_KEY_ID, and earlier keys in PERSONAL_DATA_MASTER_KEY_PREVIOUS
layerMasterKey({ keyId, key, previous })	A key you load yourself. Keys wrapped by older master keys stay readable while they are in previous.
layerMasterKeyEphemeral	A random key for tests and demos
Your own MasterKey	A KMS such as AWS KMS, Google Cloud KMS or HashiCorp Vault, which keeps the master key out of the process

A MasterKey is two functions, wrap and unwrap, so implementing one on a KMS is a call to its encrypt and decrypt endpoints.

Keep the key store apart from the events: another database or at least another client, with its own permissions and backups. A copy of the keys outlives a forget for as long as the copy is kept, so let backups of the key table expire.

layerSql creates its table on startup. To create it with your own migrations instead, turn that off and take the migrations from PersonalData.migrations, the same on SQLite and PostgreSQL and in the shape of the stores’ migrations:

const migrations = PersonalData.migrations({ table: "personal_data_keys" })
const Keys = PersonalData.layerSql({ table: "personal_data_keys", migrate: false })

Unwrapped keys stay in memory for cacheTtl, up to cacheSize subjects, 10,000 by default. Requests for a subject that isn’t in memory share one read of the store and one call to the master key, and writers racing for a new subject create a single key, so a KMS is called once rather than once per writer.

Rotating the master key

A new master key needs a new keyId, and wraps the keys of subjects created from then on. Keys of earlier subjects stay wrapped by the master key that created them, so that one goes in previous until they are moved:

1. Deploy the new master key as the current one, with the old one in previous. With layerMasterKeyConfig(), that is PERSONAL_DATA_MASTER_KEY_PREVIOUS, as id:base64 pairs separated by commas.
2. Run PersonalData.rewrap() once, from any process. It wraps every subject key with the current master key, a page at a time, and returns how many it moved. The keys themselves don’t change, so no event is rewritten, and running it again only moves what is left.
3. Drop the old key from previous.

A KMS that rotates versions of one key behind the same name needs none of this.

On Google Cloud KMS

With @google-cloud/kms, the master key is a symmetric crypto key that never leaves Cloud KMS. wrap and unwrap call its encrypt and decrypt. effect-cqrs-kms runs this code against your own Cloud KMS, with the steps to set it up, rotate the key and forget a subject:

import { KeyManagementServiceClient } from "@google-cloud/kms"
import { Config, Effect, Layer } from "effect"
import { PersonalData } from "effect-cqrs"

/**
 * Wraps subject keys with a symmetric Cloud KMS key, named like
 * `projects/acme/locations/europe-west6/keyRings/app/cryptoKeys/personal-data`.
 */
const layerGoogleKms = (keyName: string) =>
  Layer.effect(
    PersonalData.MasterKey,
    Effect.gen(function*() {
      const client = yield* Effect.acquireRelease(
        Effect.sync(() => new KeyManagementServiceClient()),
        (client) => Effect.promise(() => client.close())
      )
      // Binds a wrapped key to its subject and id, so it can't be swapped for another.
      const context = (subject: string, keyId: string) => Buffer.from(JSON.stringify([subject, keyId]))
      const call = <A>(operation: "wrap" | "unwrap", subject: string, request: () => Promise<A>) =>
        Effect.tryPromise({
          try: request,
          catch: (cause) =>
            new PersonalData.PersonalDataError({ operation, subject, message: "Cloud KMS call failed", cause })
        })
      return PersonalData.MasterKey.of({
        keyId: keyName,
        wrap: (dataKey, { keyId, subject }) =>
          call("wrap", subject, () =>
            client.encrypt({
              name: keyName,
              plaintext: dataKey,
              additionalAuthenticatedData: context(subject, keyId)
            })).pipe(Effect.map(([{ ciphertext }]) => new Uint8Array(ciphertext as Uint8Array))),
        // Keys wrapped by an earlier crypto key name it as `masterKeyId`.
        unwrap: (wrapped, { keyId, masterKeyId, subject }) =>
          call("unwrap", subject, () =>
            client.decrypt({
              name: masterKeyId,
              ciphertext: wrapped,
              additionalAuthenticatedData: context(subject, keyId)
            })).pipe(Effect.map(([{ plaintext }]) => new Uint8Array(plaintext as Uint8Array)))
      })
    })
  )

const Keys = PersonalData.layerSql().pipe(
  Layer.provide(Layer.unwrap(Effect.map(Config.String("PERSONAL_DATA_KMS_KEY"), layerGoogleKms))),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)

• Access. The client signs in with Application Default Credentials. The service account needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key, and nothing else in Cloud KMS.
• Rotation. Rotate the key in Cloud KMS as often as you like: decrypt uses the version a key was wrapped with, so nothing changes here. Moving to another crypto key only takes a new PERSONAL_DATA_KMS_KEY. Keys wrapped by the old one record its name as masterKeyId, and stay readable while the service account may still use it.
• Calls. Cloud KMS bills each operation: one encrypt for each new subject, and one decrypt each time a subject’s key isn’t in memory, see cacheTtl above.
• Forgetting deletes the subject’s wrapped key from the key store, so Cloud KMS has nothing left to decrypt. Disabling or destroying the crypto key does the opposite: every subject becomes unreadable at once.

Google also recommends checking each request’s CRC32C, with plaintextCrc32c and ciphertextCrc32c, to catch data corrupted on the way. Add those fields to the requests if you want that check.

On Node

Encryption runs on WebCrypto, which works in Node and in browsers but is asynchronous. On Node, PersonalDataNode.layerCipher encrypts with node:crypto instead, synchronously, which makes a personal field about half as expensive to read. Both write the same format, so data written with one reads with the other. It is a module of its own, so that code for browsers never imports node:crypto:

import * as PersonalDataNode from "effect-cqrs/PersonalDataNode"

const Keys = PersonalData.layerSql().pipe(
  Layer.provide(PersonalData.layerMasterKeyConfig()),
  Layer.provide(PersonalDataNode.layerCipher),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)

The cipher is a PersonalData.Cipher, provided to the keys and the master key when they are built.

Forgetting

yield* PersonalData.forget("alice")

• The key is destroyed and the subject is marked as forgotten for good. Commands that would record new personal data of the subject fail with SubjectForgotten.
• The request is recorded as a SubjectForgotten event in the stream PersonalData.streamName, with the current causation and metadata, and no personal data. Running it again records nothing new.
• States and snapshots follow. A state in memory that holds the subject’s data is rebuilt the next time it is used. Snapshots are encrypted with the key of every subject they hold data of, so the snapshot becomes unreadable and is deleted on the next read.

Other processes keep using a key they have in memory until it expires, a minute by default. Set cacheTtl on the key store to change that.

View models

View models receive events with their personal fields decrypted, so what they store is in plaintext. onForget lets one erase it:

const Owners = ViewModel.make({
  name: "owners",
  aggregates: [Account],
  handlers: {
    AccountOpened: ({ owner }, { aggregateId }) => OwnerStore.set(aggregateId, owner)
  },
  onForget: (subject) => OwnerStore.delete(subject),
  dispose: OwnerStore.clear
})

onForget runs when the SubjectForgotten event comes up, in order with the other events and again on every replay. Events after it already carry the replacements. A view model without onForget only loses the data when it is replayed.

What it doesn’t cover

• Fields marked later. Events stored before a field became personal keep it in plaintext, because the log is never rewritten. They still decode.
• Copies outside effect-cqrs. Logs, traces, caches and exports that saw a decrypted value keep it. Don’t log payloads of events with personal fields.
• Identifiers. The subject id is stored in plaintext next to each encrypted field, so use an opaque id rather than an email address.
• Deleted bytes. Databases keep overwritten rows for a while in free pages, dead tuples or their write-ahead log, until they are vacuumed.
• Copies within a stream. An encrypted field is bound to its stream, event type, place in the event and subject, so it doesn’t decrypt anywhere else. It isn’t bound to one event: someone who can write to the store can copy it into another event of the same type in the same stream.

The cost is one AES-GCM operation per personal field on every write and read, see performance.