A test dataset that looks like production does not exist. Fixtures hold two hundred evenly spread rows; production holds one customer with four million rows, three duplicate addresses and a column nobody has filled in since 2023.
This tutorial builds the third way: on every pull request, a branch forked from production, dynamic masking applied to it, the test suite running against real volumes with fake identities, and deletion when the PR closes. An hour to set up, then nothing left to do.
Between fixtures that resemble nothing and a copy of production on a developer’s machine, most teams pick the first and discover the second during an incident.
The bugs that cost money do not show up on two hundred evenly spread rows. They show up on a forty-million-row table, on a distribution where one customer accounts for 60 % of the volume, on null values nobody planned for. A synthetic dataset never produces them — it produces exactly what its author had in mind, which is precisely the problem.
The third path rests on three pieces that already exist: a Lampion branch, which is a copy-on-write fork of production at the current LSN (Log Sequence Number); the postgresql_anonymizer extension, which masks on read without rewriting a single row; and a lifetime that deletes the branch when the pull request closes.
Which columns are personal, and what to replace them with: that is a team decision, not an infrastructure setting. So it belongs in the repository, reviewed in a pull request like everything else.
[
{ "table": "users", "column": "email", "function": "anon.fake_email()" },
{ "table": "users", "column": "last_name", "function": "anon.fake_last_name()" },
{ "table": "users", "column": "phone", "function": "anon.random_phone($$0X XX XX XX XX$$)" },
{ "table": "customers", "column": "company", "function": "anon.fake_company()" },
{ "table": "customers", "column": "iban", "function": "anon.fake_iban()" },
{ "table": "orders", "column": "notes", "function": "$$REDACTED$$" }
]Versioning this file has a useful side effect: adding a personal column without adding its rule becomes visible in code review. It is also the document you will show when someone asks which data leaves for testing.
The API accepts only this list — any other value is rejected with a 400. It is an allowlist, and it is what keeps a masking rule from becoming an SQL injection vector.
| Function | What it returns | Good for |
|---|---|---|
| anon.fake_email() | A plausible, fake address | Email columns — the format stays valid |
| anon.fake_first_name() | A first name from the dictionary | First names, display columns |
| anon.fake_last_name() | A surname from the dictionary | Last names |
| anon.fake_city() | A city | Postal addresses |
| anon.fake_company() | A company name | B2B customer names |
| anon.fake_iban() | A well-formed IBAN | Bank details |
| anon.fake_siret() | A well-formed SIRET | Company identifiers |
| anon.partial({COL},2,$$***$$,2) | First 2 and last 2 characters, the rest masked | Phone numbers, references — keeps the length |
| anon.hash({COL}) | A stable SHA-256 | When equality between rows must survive |
| anon.random_string(10) | Ten random characters | Free-form fields with no imposed format |
| anon.random_zip() | A postal code | Postal codes |
| anon.random_date() | A random date | Dates of birth |
| anon.random_phone($$0X XX XX XX XX$$) | A number in French format | Phone numbers when the format matters |
| $$REDACTED$$ | A constant | Free-text comment fields |
| NULL | Nothing | Columns the tests do not need |
A copy-on-write fork at main’s current LSN: real data, in seconds, with no physical copy. Keep its identifier — everything else hangs off it.
# The PR branch, forked from main $ lampion branches create prj_a1b2c3d4 pr-482 --parent prj_a1b2c3d4 ✓ pr-482 · copy-on-write fork at current LSN · seeds replayed (3) # Its identifier: this is what the masking commands expect $ lampion branches list prj_a1b2c3d4 --json | jq -r '.[] | select(.name == "pr-482") | .id' br-9c4a71f2e08d
At this point the branch holds production data, in the clear. The copy-on-write fork shares the parent’s storage: nothing has been anonymised, nothing has even been copied. The masking in the next step acts on reads, not on content. Until it is enabled and verified, treat this branch exactly like production.
One command installs the extension and switches the compute into dynamic masking; a loop applies the rules from the repository. No data is rewritten.
# Installs the anon extension, creates the analyst role, starts dynamic masking $ lampion anon enable prj_a1b2c3d4 br-9c4a71f2e08d ✓ Anonymization enabled. lampion_analyst role created. # Applies the repository rules, one per column $ jq -c '.[]' masking.json | while read -r r; do lampion anon add-rule prj_a1b2c3d4 br-9c4a71f2e08d \ --table "$(jq -r .table <<<"$r")" \ --column "$(jq -r .column <<<"$r")" \ --function "$(jq -r .function <<<"$r")" done ✓ Rule added: public.users.email -> anon.fake_email()
What enable does, precisely
Enabling or disabling masking and managing rules require the admin role; reading the status and the rule list only needs project access. An API key inherits the role of whoever created it in the organisation — so a key made by a developer account will return 403 on anon enable. Have an admin or owner generate the CI key, and store it as a repository secret.
This is the step people skip, and the only one that proves anything. The status shown by the console says what was requested; only a read says what is in force.
# What the console believes $ lampion anon status prj_a1b2c3d4 br-9c4a71f2e08d Enabled: true id schema table column masking_function a3f9c210b74e public users email anon.fake_email()
This double check is not belt and braces. When applying a rule fails on the compute, it is still recorded on the console side and the API answers 201: the status can therefore show a rule that masks nothing. The data is the only reliable witness, and it is also the assertion to put in CI.
-- The same query, with both connection strings.
DO $$
DECLARE leaked int;
BEGIN
SELECT count(*) INTO leaked
FROM users
WHERE email LIKE '%@acme.fr'
OR email LIKE '%@beta-sa.fr';
IF leaked <> 0 THEN
RAISE EXCEPTION 'masquage inactif : % adresse(s) reelle(s) lisibles', leaked;
END IF;
END $$;The assertion to automate
Nothing changes in the test code: only the connection string does. Because masking is carried by the role, the suite sees fake identities without knowing it.
Choosing the suite’s role
# The analyst connection string, published by the endpoint once masking is on $ export DATABASE_URL=$(lampion endpoints list prj_a1b2c3d4 --json \ | jq -r '.[] | select(.name == "pr-482") | .analyst_connection_string') # Then the suite, unchanged $ npm test ✓ 214 passed · 0 failed
A pleasant consequence: tests run against the same amount of data as production, so slow queries are slow in tests too. That is the main point of the exercise, ahead of compliance.
The six previous commands, assembled: pull request opened, masking, verification, tests, then deletion on close.
# A masked production database per pull request, destroyed on close
name: masked-tests
on:
pull_request:
types: [opened, synchronize, closed]
env:
# Key generated by an admin account: anon enable requires it.
LAMPION_TOKEN: ${{ secrets.LAMPION_TOKEN }}
PROJECT_ID: prj_a1b2c3d4
BRANCH: pr-${{ github.event.number }}
jobs:
test:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install lampion-cli
- name: Fork production
run: |
lampion branches create "$PROJECT_ID" "$BRANCH" --parent "$PROJECT_ID"
BRANCH_ID=$(lampion branches list "$PROJECT_ID" --json \
| jq -r --arg b "$BRANCH" '.[] | select(.name == $b) | .id')
echo "BRANCH_ID=$BRANCH_ID" >> "$GITHUB_ENV"
- name: Mask
run: |
lampion anon enable "$PROJECT_ID" "$BRANCH_ID"
jq -c '.[]' masking.json | while read -r rule; do
lampion anon add-rule "$PROJECT_ID" "$BRANCH_ID" \
--table "$(jq -r .table <<<"$rule")" \
--column "$(jq -r .column <<<"$rule")" \
--function "$(jq -r .function <<<"$rule")"
done
- name: Fetch the masked connection string
run: |
URL=$(lampion endpoints list "$PROJECT_ID" --json \
| jq -r --arg b "$BRANCH" \
'.[] | select(.name == $b) | .analyst_connection_string')
test -n "$URL" && test "$URL" != "null"
echo "::add-mask::$URL"
echo "DATABASE_URL=$URL" >> "$GITHUB_ENV"
# Masking is proven on the data, never on the status.
- name: Verify masking
run: psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f tests/assert_masked.sql
- name: Tests
run: npm test
cleanup:
if: github.event.action == 'closed'
runs-on: ubuntu-latest
steps:
- run: pip install lampion-cli
- run: |
BRANCH_ID=$(lampion branches list "$PROJECT_ID" --json \
| jq -r --arg b "$BRANCH" '.[] | select(.name == $b) | .id')
lampion branches delete "$PROJECT_ID" "$BRANCH_ID"The verification job comes before the tests on purpose: if masking is not in place, the suite must not run at all. And since rules are not inherited when a branch is created, they are reapplied every time — tedious by hand, free in a workflow.
Since the branch carries real volumes, you may as well compare the heavy queries between production and the pull request.
# Compare the expensive queries between two endpoints $ curl -X POST "$LAMPION_API/v1/projects/prj_a1b2c3d4/replay" \ -H "Authorization: Bearer $LAMPION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"source_endpoint_id": "ep-4f9c21ab8d3e", "target_endpoint_id": "ep-7d31c8f0a94b", "limit": 20}' {"summary": {"total": 20, "regressions": 1, "improvements": 3, "stable": 12}}
Let us name it properly: this is not a replay of production traffic, it is a benchmark of heavy non-parameterised queries. Useful for catching an index dropped by accident or a join gone quadratic; useless for predicting behaviour under load. There is no CLI command for this call — it is the API or the console.
What this comparator really does
Masking rewrites the read, not the storage. A masked branch is still a logical copy of production: deleting it is part of the mechanism, not the housekeeping.
Ten checks. The first four prevent a leak, the next six prevent false confidence.
Free: 3 projects, 3 branches, 0.25 CU, 512 MB of storage — no credit card. Data hosted in France.