Backup and restore

What this page covers

The backup scope narrows to one folder root plus one environment file. This page lists what to back up, how to do it, how to restore, and the ten things to verify afterwards before a restore counts. Backups use pg_dump, pg_restore, and tar, which come with the database and the operating system.

What you need

Three persistent locations, all bind mounts:

Host location	Content
${DATA_PATH}/postgres	All business data, audit records, and encrypted credentials and key wrappings
${DATA_PATH}/recordings	Session recording files
${DATA_PATH}/audit	Audit downgrade files and sealing period logs

Plus .env, which carries the key configuration. Keep backups encrypted, and away from the master key material. Directory permissions on DATA_PATH should be 0750 or stricter.

The export directory and the offsite staging area fall outside the backup scope: an evidence bundle has a retention period of its own and is cleared when it ends, and the staging area holds nothing worth keeping once the upload completes.

How to set it up

For a package deployment with the built-in database, choose Backup in the custodexa.sh menu or run sudo /opt/custodexa/custodexa.sh backup. The manual procedure below assumes the built-in database and local postgres directory. With an external database, arrange a consistent backup of that database, data files, .env, and tls/ yourself. Export artifacts persist at ${DATA_PATH}/exports, outside the default backup scope; download items needed beyond their retention period.

0. Read the variables first

Every procedure starts the same way. Read .env literally rather than sourcing it:

ENV_FILE="${ENV_FILE:-./.env}"
env_get() { sed -n "s/^[[:space:]]*$1=//p" "$ENV_FILE" | tail -n 1; }
DATA_PATH="${DATA_PATH:-$(env_get DATA_PATH)}"
DB_USER="${DB_USER:-$(env_get DB_USER)}"
DB_NAME="${DB_NAME:-$(env_get DB_NAME)}"
printf 'DATA_PATH=%s\nDB_USER=%s\nDB_NAME=%s\n' "$DATA_PATH" "$DB_USER" "$DB_NAME"

Refer to them afterwards as "${VAR:?}", so a missing value fails on the spot.

1. Cold backup

The backup before an upgrade has to be a cold one. The order is stop the application, back up the database, back up the files, back up the configuration, then start again:

1. docker compose stop backend guacd frontend
2. Export the database with pg_dump -Fc
3. Pack the recordings and audit directories with tar -czf
4. Copy .env
5. docker compose start backend guacd frontend
6. Confirm both artifacts are readable with pg_restore --list and tar -tzf

2. Hot backup

A routine backup can run hot: skip the stop and start steps, back up the database first, then copy the recording directory. The direction of the inconsistency is deliberate; better that the recording directory be newer than the database.

How to restore

The order differs from the backup in one place: .env is restored first, and the variables are read afterwards:

1. docker compose down
2. Restore .env
3. Read the variables from the restored .env
4. Unpack the recording and audit archives (the target postgres directory has to be empty)
5. Start the database service alone and wait until it accepts connections
6. Restore the database with pg_restore --clean --if-exists
7. On Linux, set the recordings directory ownership to 1000:0 and mode to 2770, then run docker compose up -d to start everything

Ten checks after a restore

1. Every service is up.
2. The backend health check passes.
3. The startup log holds no fatal error.
4. The frontend returns 200.
5. The login path works.
6. The four fingerprints on the key inventory match the ones from the backup.
7. Encrypted fields decrypt (open an asset’s editor and confirm the credential still works).
8. A recording picked at random plays back.
9. The audit chain verifies.
10. On deployments with offsite storage, reconcile the ledger against the bucket.

Suggested settings for the object storage bucket

The offsite bucket is managed by the deployer. Suggested:

• S3 or a compatible service: turn on versioning; turn on object lock at bucket creation when you want to prevent deletion and modification (it cannot be added afterwards); align the expiry days of the lifecycle rule with the recording retention policy, and add an expiry rule for noncurrent versions; narrow the access permissions to writing under the given prefix, reading objects and their metadata, and reading the bucket configuration.
• GCS: use the bucket retention policy as the baseline, and grant permissions at the level of object creation and object viewing.

What auditors can see

• The chain verification result after a restore, issued per interval, shows that the restored audit data is still self-consistent.
• Comparing key fingerprints is direct evidence that the cryptographic material was not replaced.
• The offsite ledger records a hash and custody chain events per item; for the reconciliation, see Offsite evidence storage.