Upgrade and rollback

What this page covers

Keep a cold backup and verified previous images before upgrading, then check services and the evidence chain. Package deployments use custodexa.sh upgrade. See Backup and restore for backup steps.

What you need

• A pre-upgrade cold backup, previous release package, and verified image IDs.
• Records of .env, the data directory, and four key fingerprints.
• A maintenance window for existing sessions and database migrations.

How to set it up

Before upgrading

Stop new connections and wait for existing sessions and the audit queue to drain. Deployments with the built-in database can run custodexa.sh backup or follow Backup and restore for a cold backup. With an external database, arrange a consistent backup of that database, data files, .env, and tls/ yourself. Record the version, image IDs, key fingerprints, and table counts. Use an absolute DATA_PATH.

New export artifacts persist at ${DATA_PATH}/exports, outside the default backup scope. Before manually upgrading an older version, move any needed artifacts out of the container or download them.

Package deployment

Use the installed custodexa.sh to query for updates, choose a target in the menu, or run:

sudo /opt/custodexa/custodexa.sh upgrade             # query only
sudo /opt/custodexa/custodexa.sh upgrade 1.14.0      # target version
sudo /opt/custodexa/custodexa.sh status

A downloaded package path is also accepted. A query without a target does not accept --images-from. With an installed 1.14.0 or later script, the menu offers an image source; a command may use --images-from auto|source, defaulting to auto. Auto checks this host, an offline bundle, GHCR, Docker Hub, then a local build. Source verifies the package source before building backend and frontend; upstream images and build dependencies still need to be obtained, so it is not fully offline. Downloads, checks and builds show progress before they run and a result after they finish.

For the first upgrade from 1.13.x to 1.14.0, the installed old script starts the upgrade. It does not recognize --images-from and its menu has no source choice. Omit the flag; the 1.14.0 target uses Auto. Later upgrades can select source building after 1.14.0 is installed. The script previews downtime and backs up its own database; arrange a consistent external database backup yourself. There is no automatic rollback. Check the backend startup log for the database migration result.

Manual migration of an older git clone deployment

Running scripts/custodexa/custodexa.sh inside an old git clone, or pointing CUSTODEXA_HOME at it, explains that this is an older source deployment, leaves deployment files and services unchanged, and refuses the operation. Check backups and deployment settings first. Follow the manual migration section of the Upgrade SOP, or install the package in a separate clean directory and restore by hand using Backup and Restore. Do not run install against the existing data directory.

Verify and roll back

Check health, key fingerprints, the checkpoint chain, playback of an older recording, and auditing of a new connection. For rollback, stop services and restore the pre-upgrade database, .env, recordings, and audit directory with the old package and verified images. Repeat verification. Database migrations are not reversed.

What auditors can see

Checkpoint verification and old and new sessions let reviewers check the evidence chain. The recording is the source of truth for text sessions, and input without echo leaves no command-text record. The query console’s structured statement record is the source of truth for that query.