A test dataset that looks like production does not exist. Fixtures hold two hundred evenly spread rows; production holds one customer with four million rows, three duplicate addresses and a column nobody has filled in since 2023.
This tutorial builds a middle way: on every pull request, a branch forked from production, dynamic masking applied to it, the test suite running against real volumes while reading only fake values, and deletion when the PR closes. To be clear from the start: the real data stays in the branch, only reading it is masked. This is not anonymisation in the GDPR sense — at best pseudonymisation, which reduces the exposure of personal data in testing without making the branch anything other than processing of that data.
Between fixtures that resemble nothing and a copy of production on a developer’s machine, most teams pick the first and discover the second during an incident.
The bugs that cost money do not show up on two hundred evenly spread rows. They show up on a forty-million-row table, on a distribution where one customer accounts for 60 % of the volume, on null values nobody planned for. A synthetic dataset never produces them — it produces exactly what its author had in mind, which is precisely the problem.
The third path rests on three pieces that already exist: a Lampion branch, which is a copy-on-write fork of production at the current LSN (Log Sequence Number); the postgresql_anonymizer extension, which masks on read without rewriting a single row; and a bounded lifetime, with deletion when the pull request closes and a TTL (Time To Live) as a safety net.
Which columns are personal, and what to replace them with: that is a team decision, not an infrastructure setting. So it belongs in the repository, reviewed in a pull request like everything else.
[
{ "table": "users", "column": "email", "function": "anon.fake_email()" },
{ "table": "users", "column": "last_name", "function": "anon.fake_last_name()" },
{ "table": "users", "column": "phone", "function": "anon.random_phone($$0X XX XX XX XX$$)" },
{ "table": "customers", "column": "company", "function": "anon.fake_company()" },
{ "table": "customers", "column": "iban", "function": "anon.fake_iban()" },
{ "table": "orders", "column": "notes", "function": "$$REDACTED$$" }
]Versioning this file has a useful side effect: adding a personal column without adding its rule becomes visible in code review. It is also the document you will show when someone asks which data leaves for testing.
The API accepts only this list — any other value is rejected with a 400. It is an allowlist, and it is what keeps a masking rule from becoming an SQL injection vector.
| Function | What it returns | Good for |
|---|---|---|
| anon.fake_email() | A plausible, fake address | Email columns — the format stays valid |
| anon.fake_first_name() | A first name from the dictionary | First names, display columns |
| anon.fake_last_name() | A surname from the dictionary | Last names |
| anon.fake_city() | A city | Postal addresses |
| anon.fake_company() | A company name | B2B customer names |
| anon.fake_iban() | A well-formed IBAN | Bank details |
| anon.fake_siret() | A well-formed SIRET | Company identifiers |
| anon.partial({COL},2,$$***$$,2) | First 2 and last 2 characters, the rest masked | Phone numbers, references — keeps the length |
| anon.hash({COL}) | A stable digest | When equality between rows must survive |
| anon.random_string(10) | Ten random characters | Free-form fields with no imposed format |
| anon.random_zip() | A postal code | Postal codes |
| anon.random_date() | A random date | Dates of birth |
| anon.random_phone($$0X XX XX XX XX$$) | A number in French format | Phone numbers when the format matters |
| $$REDACTED$$ | A constant | Free-text comment fields |
| NULL | Nothing | Columns the tests do not need |
A copy-on-write fork at main’s current LSN: real data, in seconds, with no physical copy. The name is unique within the project — creating it a second time returns 409. Keep its identifier — everything else hangs off it.
# The PR branch, forked from main (for --parent, the project ID means main), deleted automatically after 72 h $ export PROJECT_ID=3f8a2c1e9b7d4f60a5e2c8d1b4f7a903 $ export BRANCH_ID=$(lampion branches create "$PROJECT_ID" pr-482 \ --parent "$PROJECT_ID" --ttl 72 --json | jq -r .id) # Its identifier: this is what every following command expects $ echo "$BRANCH_ID" 9c4a71f2e08d4b3a8f61d2c7e5b09a14
At this point the branch holds production data, in the clear. The copy-on-write fork shares the parent’s storage: nothing has been masked, nothing has even been copied. The masking in the next step acts on reads, not on content. Until it is enabled and verified, treat this branch exactly like production — and even afterwards, it remains a copy of it.
One command installs the extension and switches the compute into dynamic masking; a loop applies the rules from the repository. No data is rewritten.
# Installs the anon extension, creates the analyst role, starts dynamic masking $ lampion anon enable "$PROJECT_ID" "$BRANCH_ID" Anonymization enabled. lampion_analyst role created. # Applies the repository rules, one per column $ jq -c '.[]' masking.json | while read -r r; do lampion anon add-rule "$PROJECT_ID" "$BRANCH_ID" \ --table "$(jq -r .table <<<"$r")" \ --column "$(jq -r .column <<<"$r")" \ --function "$(jq -r .function <<<"$r")" done Rule added: public.users.email -> anon.fake_email()
What enable does, precisely
Enabling or disabling masking and managing rules require the admin role; reading the status and the rule list only needs project access. An API key inherits the role of whoever created it in the organisation — so a key made by a developer account will return 403 on anon enable. Have an admin or owner generate the CI key, and store it as a repository secret.
This is the step people skip, and the only one that proves anything. The status shown by the console says what was requested; only a read says what is in force.
# What the console believes (abridged output) $ lampion anon status "$PROJECT_ID" "$BRANCH_ID" Enabled: True id schema_name table_name column_name masking_function a3f9c210b74e public users email anon.fake_email()
This double check is not belt and braces. A rule that fails to apply on the compute is now rejected (502) and not recorded — for a long time it was recorded anyway, with a 201 and a listed rule that masked nothing. But the status remains a declaration: it sees neither a personal column missing from masking.json nor a suite wired to the wrong role. The data is the only reliable witness, and it is the assertion to put in CI.
-- Run as the role your tests use (here lampion_analyst), never as cloud_admin.
-- The first error stops the script, and psql exits with a failure.
\set ON_ERROR_STOP on
DO $$
DECLARE
total bigint;
leaked bigint;
BEGIN
-- 0. An empty table proves nothing.
SELECT count(*) INTO total FROM users;
IF total = 0 THEN
RAISE EXCEPTION 'table users is empty: nothing to verify';
END IF;
-- 1. No address may belong to a known customer domain.
SELECT count(*) INTO leaked
FROM users
WHERE lower(split_part(email, '@', 2)) IN ('acme.fr', 'beta-sa.fr');
IF leaked > 0 THEN
RAISE EXCEPTION 'masking inactive: % customer-domain address(es) readable in the clear', leaked;
END IF;
-- 2. Canary row, created on purpose in production: its real value must never come back.
IF EXISTS (SELECT 1 FROM customers WHERE company = 'Temoin CI SAS') THEN
RAISE EXCEPTION 'masking inactive: the canary row comes back in the clear';
END IF;
END $$;The assertion to automate
Nothing changes in the test code: only the connection string does. Because masking is carried by the role, the suite reads fake identities without knowing it.
Choosing the suite’s role
# The analyst connection string, published by the endpoint once masking is on $ export DATABASE_URL=$(lampion endpoints list "$PROJECT_ID" --json \ | jq -r --arg id "$BRANCH_ID" '.[] | select(.branch_id == $id) | .analyst_connection_string') # Then the suite, unchanged $ npm test ✓ 214 passed · 0 failed
A pleasant consequence: tests run against the same amount of data as production, so slow queries are slow in tests too. That is the main point of the exercise, ahead of compliance.
The previous steps, assembled: one branch per pull request, created on the first run and reused on the next ones, masking, verification, tests, then deletion on close.
# A masked production database per pull request, destroyed on close
name: masked-tests
on:
pull_request:
types: [opened, reopened, synchronize, closed]
# One run at a time per pull request: two pushes in quick succession
# do not create two branches.
concurrency:
group: ${{ github.workflow }}-${{ github.event.number }}
defaults:
run:
shell: bash # bash -eo pipefail: any failing command stops the step
env:
# Key generated by an admin account: anon enable requires it.
LAMPION_TOKEN: ${{ secrets.LAMPION_TOKEN }}
PROJECT_ID: 3f8a2c1e9b7d4f60a5e2c8d1b4f7a903
BRANCH: pr-${{ github.event.number }}
jobs:
test:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install "lampion-cli>=0.2.1"
# Created on the first run, found by name on later pushes.
- name: Fork production
run: |
BRANCH_ID=$(lampion branches list "$PROJECT_ID" --json \
| jq -r --arg b "$BRANCH" '.[] | select(.name == $b) | .id')
if [ -z "$BRANCH_ID" ]; then
# --parent = the project ID, i.e. main; --ttl = safety net if cleanup never runs.
BRANCH_ID=$(lampion branches create "$PROJECT_ID" "$BRANCH" \
--parent "$PROJECT_ID" --ttl 72 --json | jq -r .id)
echo "NEW_BRANCH=true" >> "$GITHUB_ENV"
fi
test -n "$BRANCH_ID" && test "$BRANCH_ID" != "null"
echo "BRANCH_ID=$BRANCH_ID" >> "$GITHUB_ENV"
# main is not masked: the branch inherits no rules, so we apply them.
- name: Mask
if: env.NEW_BRANCH == 'true'
run: |
lampion anon enable "$PROJECT_ID" "$BRANCH_ID"
jq -c '.[]' masking.json | while read -r rule; do
lampion anon add-rule "$PROJECT_ID" "$BRANCH_ID" \
--table "$(jq -r .table <<<"$rule")" \
--column "$(jq -r .column <<<"$rule")" \
--function "$(jq -r .function <<<"$rule")"
done
- name: Fetch the masked connection string
run: |
URL=$(lampion endpoints list "$PROJECT_ID" --json \
| jq -r --arg id "$BRANCH_ID" \
'.[] | select(.branch_id == $id) | .analyst_connection_string')
test -n "$URL" && test "$URL" != "null"
echo "::add-mask::$URL"
echo "DATABASE_URL=$URL" >> "$GITHUB_ENV"
# Masking is proven on the data, never on the status.
- name: Verify masking
run: psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f tests/assert_masked.sql
- name: Tests
id: tests
run: npm test
# Failure before the tests (masking missing or unproven): the branch is
# deleted, and the next push starts again from a fresh fork.
- name: Delete a badly masked branch
if: failure() && env.BRANCH_ID != '' && steps.tests.outcome == 'skipped'
run: lampion branches delete "$PROJECT_ID" "$BRANCH_ID"
cleanup:
if: github.event.action == 'closed'
runs-on: ubuntu-latest
steps:
- run: pip install "lampion-cli>=0.2.1"
- name: Delete the pull request branch
run: |
BRANCH_ID=$(lampion branches list "$PROJECT_ID" --json \
| jq -r --arg b "$BRANCH" '.[] | select(.name == $b) | .id')
if [ -n "$BRANCH_ID" ]; then
lampion branches delete "$PROJECT_ID" "$BRANCH_ID"
fiVerification runs before the tests: if masking is not proven, the suite does not run, and the branch is deleted so the next push starts from a fresh fork. Masking is applied once, when the branch is created. A branch created from a masked parent inherits its status and rules; but here the parent is main, which is not masked and cannot be — so the workflow applies the rules itself, which is tedious by hand and free in a workflow.
Since the branch carries real volumes, a comparator — through the API or the console, with no CLI command — replays main’s most expensive queries against it.
# Compare the expensive queries between main’s endpoint and the PR’s $ curl -X POST "https://api.lampion.cloud/v1/projects/$PROJECT_ID/replay" \ -H "Authorization: Bearer $LAMPION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"source_endpoint_id": "ep-3f8a2c1e9b7d", "target_endpoint_id": "ep-9c4a71f2e08d", "limit": 20}' # Abridged response: per-query details are in "queries" {"summary": {"total": 20, "regressions": 1, "improvements": 3, "stable": 12}}
It runs once on the target each of the source’s queries with the highest cumulative time (pg_stat_statements): above ×2, a regression; below ×0.5, an improvement. Parameterised queries are skipped, with no concurrency and no real ordering: it is a comparison bench, not a traffic replay.
Masking rewrites the read, not the storage. A masked branch is still a logical copy of production: deleting it is part of the mechanism, not the housekeeping.
Ten checks. The first four prevent a leak, the next six prevent false confidence.
Free: 3 projects, 3 branches, 0.25 CU, 512 MB of storage — no credit card. Data hosted in France.