Stop any further deploys, take a backup of the database exactly as it is now, and find out how far the migration got. A partly applied migration is the dangerous kind. Then there are two ways forward: roll back what ran, fix the migration and deploy again, or finish the remaining steps by hand and mark the migration as resolved. Never "fix" it with a command that resets or recreates the database. That deletes your data.
What's the problem
A deploy ran a database migration (a script that changes the tables) and it failed. The app may be down, half-working, or stuck in a loop of failed deploys. The migration tool refuses to run anything else until this one is dealt with.
Why it happens
- Production data is different. A migration that added a required column, or a unique rule, worked on an empty development database and failed on real rows that break the rule.
- Production is a different database. A different engine version or settings, so SQL that worked locally doesn't run.
- It took too long. Changing a large table can lock it or time out on a busy production database.
- The history doesn't match. Migrations were edited after being applied, applied out of order, or someone changed the database by hand.
How to fix it
- Pause deploys, so nothing retries the migration automatically.
- Back up the database now, before changing anything.
- Find exactly what ran. Your migration tool records what's been applied. Compare that with the migration's steps, and check the database to see which changes actually exist.
- Read the error to find the root cause: a constraint the data breaks, a syntax difference, a timeout.
- Choose a way forward. Either undo the parts that ran, fix the migration (for example, fill in the missing values before making a column required), mark it rolled back and redeploy. Or finish the remaining steps by hand and mark the migration as applied.
- Test the fix on a copy of production data before running it for real.
- Bring the app back up and check the flows that use the changed tables.
When to call Preventionlabs
If the error is clear and the fix is one step, you can do this carefully. Call us when nobody knows the state of the database, the migration history has drifted from the code, or the app has been down while people guess. A resurrection gets the app running, with the user flows you list working end to end on its real data, in your own hosting account. Backups remain your responsibility throughout, as our terms say, so take one before anyone touches it.
Submit your project for a free assessmentFree assessment. $10,000 AUD flat to get it live, only if we take it on and you go ahead.
Sources
- Prisma: Patching & hotfixing: failed migrationsdatabase tool docs
There are two ways to deal with failed migrations in a production environment:
- Prisma: Patching & hotfixing: failed migrationsdatabase tool docs
Fix the root cause of the failed migration, if relevant - for example, if the migration failed due to an issue with the SQL script itself.
- The Twelve-Factor App: X. Dev/prod parityengineering methodology
Differences between backing services mean that tiny incompatibilities crop up, causing code that worked and passed tests in development or staging to fail in production.