# Personal data

> Forget a person without rewriting the log, by encrypting their data with a key you can destroy.

The log is append-only, so an event can’t be edited or deleted when someone asks to be forgotten, as the GDPR allows. Crypto-shredding gets around that: the personal fields of an event are encrypted with a key that belongs to the person they describe, the *subject*, and forgetting the subject destroys the key. The event stays where it is with its amounts, dates and ids, so streams still replay and balances still add up, while its personal fields read as a replacement.

## Marking fields

`PersonalData.field` marks a field of an event and says what it reads as once its subject is forgotten:

```ts
import { PersonalData } from "effect-cqrs"


const AccountEvent = Schema.TaggedUnion({
  AccountOpened: {
    owner: PersonalData.field(Schema.String, { redacted: "[forgotten]" }),
    email: Schema.optional(PersonalData.field(Schema.String, { redacted: "forgotten@example.invalid" })),
    currency: Schema.Literals(["CHF", "EUR"])
  },
  TransferSent: {
    to: Schema.String,
    beneficiary: PersonalData.field(Schema.String, { redacted: "[forgotten]", subject: "to" }),
    amount: Schema.Number
  }
})
```

* **The subject** is the aggregate id of the stream, unless `subject` names a property next to the field. Above, `beneficiary` belongs to the account in `to`, so forgetting the receiver of a transfer leaves the sender’s events readable.
* **The replacement** has to be a valid value of the field. It is checked when the field is defined.
* **Where a field can be:** a property of the event, of a nested struct, optional or nullable. Building an aggregate fails when a personal field sits in an array, a record or an untagged union, is transformed after being marked, or names a subject that isn’t next to it or isn’t always a string. Mark the transformed schema instead: `PersonalData.field(Schema.String.pipe(Schema.decodeTo(...)), ...)`.

Schemas stay plain: a personal field is an ordinary string to projections, handlers and tests. Encryption happens where `AggregateService` writes to and reads from the store, so nothing else can write a personal field unencrypted.

## Keys

An aggregate with personal fields needs `PersonalData.Keys`, and building its service fails without it. Each subject gets its own AES-256 key the first time it is written. The key store only holds that key wrapped by a `MasterKey`:

```ts
const Keys = PersonalData.layerSql().pipe(
  Layer.provide(PersonalData.layerMasterKeyConfig()),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)
```

| Key store                          | For                                                                                            |
| ---------------------------------- | ---------------------------------------------------------------------------------------------- |
| `PersonalData.layerMemory()`       | Tests and the browser. Keys are gone after a restart, and with them everything they encrypted. |
| `PersonalData.layerSql({ table })` | SQLite or PostgreSQL, through any `effect/sql` client                                          |

| Master key                                 | For                                                                                                                                                       |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `layerMasterKeyConfig()`                   | 32 bytes in base64 from `PERSONAL_DATA_MASTER_KEY`, with its id in `PERSONAL_DATA_MASTER_KEY_ID`, and earlier keys in `PERSONAL_DATA_MASTER_KEY_PREVIOUS` |
| `layerMasterKey({ keyId, key, previous })` | A key you load yourself. Keys wrapped by older master keys stay readable while they are in `previous`.                                                    |
| `layerMasterKeyEphemeral`                  | A random key for tests and demos                                                                                                                          |
| Your own `MasterKey`                       | A KMS such as AWS KMS, [Google Cloud KMS](#on-google-cloud-kms) or HashiCorp Vault, which keeps the master key out of the process                         |

A `MasterKey` is two functions, `wrap` and `unwrap`, so implementing one on a KMS is a call to its encrypt and decrypt endpoints.

Keep the key store apart from the events: another database or at least another client, with its own permissions and backups. A copy of the keys outlives a forget for as long as the copy is kept, so let backups of the key table expire.

`layerSql` creates its table on startup. To create it with your own migrations instead, turn that off and take the migrations from `PersonalData.migrations`, the same on SQLite and PostgreSQL and in the shape of the stores’ `migrations`:

```ts
const migrations = PersonalData.migrations({ table: "personal_data_keys" })
const Keys = PersonalData.layerSql({ table: "personal_data_keys", migrate: false })
```

Unwrapped keys stay in memory for `cacheTtl`, up to `cacheSize` subjects, 10,000 by default. Requests for a subject that isn’t in memory share one read of the store and one call to the master key, and writers racing for a new subject create a single key, so a KMS is called once rather than once per writer.

### Rotating the master key

A new master key needs a new `keyId`, and wraps the keys of subjects created from then on. Keys of earlier subjects stay wrapped by the master key that created them, so that one goes in `previous` until they are moved:

1. Deploy the new master key as the current one, with the old one in `previous`. With `layerMasterKeyConfig()`, that is `PERSONAL_DATA_MASTER_KEY_PREVIOUS`, as `id:base64` pairs separated by commas.
2. Run `PersonalData.rewrap()` once, from any process. It wraps every subject key with the current master key, a page at a time, and returns how many it moved. The keys themselves don’t change, so no event is rewritten, and running it again only moves what is left.
3. Drop the old key from `previous`.

A [KMS](#on-google-cloud-kms) that rotates versions of one key behind the same name needs none of this.

### On Google Cloud KMS

With [`@google-cloud/kms`](https://www.npmjs.com/package/@google-cloud/kms), the master key is a symmetric crypto key that never leaves Cloud KMS. `wrap` and `unwrap` call its `encrypt` and `decrypt`. [effect-cqrs-kms](https://github.com/davidnussio/effect-cqrs-kms) runs this code against your own Cloud KMS, with the steps to set it up, rotate the key and forget a subject:

```ts
import { KeyManagementServiceClient } from "@google-cloud/kms"
import { Config, Effect, Layer } from "effect"
import { PersonalData } from "effect-cqrs"


/**
 * Wraps subject keys with a symmetric Cloud KMS key, named like
 * `projects/acme/locations/europe-west6/keyRings/app/cryptoKeys/personal-data`.
 */
const layerGoogleKms = (keyName: string) =>
  Layer.effect(
    PersonalData.MasterKey,
    Effect.gen(function*() {
      const client = yield* Effect.acquireRelease(
        Effect.sync(() => new KeyManagementServiceClient()),
        (client) => Effect.promise(() => client.close())
      )
      // Binds a wrapped key to its subject and id, so it can't be swapped for another.
      const context = (subject: string, keyId: string) => Buffer.from(JSON.stringify([subject, keyId]))
      const call = <A>(operation: "wrap" | "unwrap", subject: string, request: () => Promise<A>) =>
        Effect.tryPromise({
          try: request,
          catch: (cause) =>
            new PersonalData.PersonalDataError({ operation, subject, message: "Cloud KMS call failed", cause })
        })
      return PersonalData.MasterKey.of({
        keyId: keyName,
        wrap: (dataKey, { keyId, subject }) =>
          call("wrap", subject, () =>
            client.encrypt({
              name: keyName,
              plaintext: dataKey,
              additionalAuthenticatedData: context(subject, keyId)
            })).pipe(Effect.map(([{ ciphertext }]) => new Uint8Array(ciphertext as Uint8Array))),
        // Keys wrapped by an earlier crypto key name it as `masterKeyId`.
        unwrap: (wrapped, { keyId, masterKeyId, subject }) =>
          call("unwrap", subject, () =>
            client.decrypt({
              name: masterKeyId,
              ciphertext: wrapped,
              additionalAuthenticatedData: context(subject, keyId)
            })).pipe(Effect.map(([{ plaintext }]) => new Uint8Array(plaintext as Uint8Array)))
      })
    })
  )


const Keys = PersonalData.layerSql().pipe(
  Layer.provide(Layer.unwrap(Effect.map(Config.String("PERSONAL_DATA_KMS_KEY"), layerGoogleKms))),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)
```

* **Access.** The client signs in with Application Default Credentials. The service account needs `roles/cloudkms.cryptoKeyEncrypterDecrypter` on the key, and nothing else in Cloud KMS.
* **Rotation.** Rotate the key in Cloud KMS as often as you like: `decrypt` uses the version a key was wrapped with, so nothing changes here. Moving to another crypto key only takes a new `PERSONAL_DATA_KMS_KEY`. Keys wrapped by the old one record its name as `masterKeyId`, and stay readable while the service account may still use it.
* **Calls.** Cloud KMS bills each operation: one `encrypt` for each new subject, and one `decrypt` each time a subject’s key isn’t in memory, see `cacheTtl` above.
* **Forgetting** deletes the subject’s wrapped key from the key store, so Cloud KMS has nothing left to decrypt. Disabling or destroying the crypto key does the opposite: every subject becomes unreadable at once.

Google also recommends checking each request’s CRC32C, with `plaintextCrc32c` and `ciphertextCrc32c`, to catch data corrupted on the way. Add those fields to the requests if you want that check.

### On Node

Encryption runs on WebCrypto, which works in Node and in browsers but is asynchronous. On Node, `PersonalDataNode.layerCipher` encrypts with `node:crypto` instead, synchronously, which makes a personal field about half as expensive to read. Both write the same format, so data written with one reads with the other. It is a module of its own, so that code for browsers never imports `node:crypto`:

```ts
import * as PersonalDataNode from "effect-cqrs/PersonalDataNode"


const Keys = PersonalData.layerSql().pipe(
  Layer.provide(PersonalData.layerMasterKeyConfig()),
  Layer.provide(PersonalDataNode.layerCipher),
  Layer.provide(SqliteClient.layer({ filename: "./data/keys.sqlite" }))
)
```

The cipher is a `PersonalData.Cipher`, provided to the keys and the master key when they are built.

## Forgetting

```ts
yield* PersonalData.forget("alice")
```

* **The key is destroyed** and the subject is marked as forgotten for good. Commands that would record new personal data of the subject fail with `SubjectForgotten`.
* **The request is recorded** as a `SubjectForgotten` event in the stream `PersonalData.streamName`, with the current causation and metadata, and no personal data. Running it again records nothing new.
* **States and snapshots follow.** A state in memory that holds the subject’s data is rebuilt the next time it is used. Snapshots are encrypted with the key of every subject they hold data of, so the snapshot becomes unreadable and is deleted on the next read.

Other processes keep using a key they have in memory until it expires, a minute by default. Set `cacheTtl` on the key store to change that.

## View models

View models receive events with their personal fields decrypted, so what they store is in plaintext. `onForget` lets one erase it:

```ts
const Owners = ViewModel.make({
  name: "owners",
  aggregates: [Account],
  handlers: {
    AccountOpened: ({ owner }, { aggregateId }) => OwnerStore.set(aggregateId, owner)
  },
  onForget: (subject) => OwnerStore.delete(subject),
  dispose: OwnerStore.clear
})
```

`onForget` runs when the `SubjectForgotten` event comes up, in order with the other events and again on every replay. Events after it already carry the replacements. A view model without `onForget` only loses the data when it is replayed.

## What it doesn’t cover

* **Fields marked later.** Events stored before a field became personal keep it in plaintext, because the log is never rewritten. They still decode.
* **Copies outside effect-cqrs.** Logs, traces, caches and exports that saw a decrypted value keep it. Don’t log payloads of events with personal fields.
* **Identifiers.** The subject id is stored in plaintext next to each encrypted field, so use an opaque id rather than an email address.
* **Deleted bytes.** Databases keep overwritten rows for a while in free pages, dead tuples or their write-ahead log, until they are vacuumed.
* **Copies within a stream.** An encrypted field is bound to its stream, event type, place in the event and subject, so it doesn’t decrypt anywhere else. It isn’t bound to one event: someone who can write to the store can copy it into another event of the same type in the same stream.

The cost is one AES-GCM operation per personal field on every write and read, see [performance](https://effect-cqrs.dambox.ch/reference/performance/).