Upgrade recovery
Customer Portal currently ships from pinned source commits rather than versioned releases. Every upgrade can change application code, dependencies, environment expectations, and the shared root migration history. Record the old and target commits before release and keep a tested database recovery point.
Identify the failure class
| Failure | Typical evidence | First response |
|---|---|---|
| Migration fails before startup | the entrypoint exits after Migration failed! | stop retrying, preserve logs, inspect database state on a restored copy |
| Migration succeeds but Nuxt fails | migration completed, process then exits or never becomes ready | preserve the migrated database, compare runtime config and target build |
| Application regression | service starts but a workflow, permission, or integration fails | stop rollout, limit traffic, determine whether the schema remains compatible with the old application |
| Data or authorization defect | incorrect records or cross-tenant exposure are possible | restrict access, preserve evidence, invoke the security process when confidentiality may be affected |
Do not assume redeploying the old image is a complete rollback. An applied migration remains applied, and old code may not understand the new schema.
Contain and diagnose
- Stop additional replicas and automated retries from changing the same database.
- Record timestamps, source and image identifiers, migration output, and the last known healthy revision.
- Take a new database snapshot before attempting repair, even if the state is partially migrated.
- Determine which migration statements committed and whether the application served traffic afterward.
- Reproduce with a copy of the pre-upgrade backup and the target commit.
- Decide among application rollback, a forward repair, or database restore.
Never edit the migration journal merely to make a migration appear complete. Never drop a feature schema as a generic rollback: removing a layer intentionally leaves its production data in place.
Choose a recovery path
Roll back application code
Use the previous image only when the new schema is demonstrably backward-compatible. Expand-and-contract migrations are designed to make this possible: old structures remain available while the new application begins using additive structures.
Run the old revision against a restored copy of the current database first. If it reads or writes removed columns, changed constraints, or transformed data incorrectly, use another path.
Repair forward
Prefer a reviewed forward migration when production has already accepted writes on the new schema or a restore would exceed the recovery objective. The repair should be generated and committed with the application change, rehearsed on a copy, and applied once through the normal release mechanism.
Restore the database
Restore the last verified recovery point into a new empty database, start the exact application revision recorded with that backup, and complete the restore validation checklist. Keep the failed database isolated for diagnosis. Cut traffic over only after application, data, and authorization checks pass.
A restore loses writes after its recovery point. The incident owner must compare that loss with the agreed RPO and coordinate any reconciliation before reopening writes.
Recover configuration separately
Database recovery does not restore hosting configuration. Compare environment variables, public URLs, OAuth callbacks, proxy settings, DNS, and email-provider credentials with the last known healthy inventory. Roll back configuration changes independently and rotate any secret whose handling became uncertain.
Close the incident
After service is stable:
- document the trigger, impact, timeline, and chosen recovery path;
- reconcile writes or external emails around the recovery window;
- add a test that would have caught the failure;
- improve the migration, release gate, monitor, or runbook;
- perform a fresh backup and schedule another recovery drill.
The preventive workflow is documented in compatibility and releases and deployment.