Backup and restore

Last reviewed: 2026-06-26

This runbook covers Rangefront Waste production and local-dev recovery. It is for operators restoring a tenant-safe wastewater SaaS environment, not for tenant data exports or ad-hoc support extracts.

Scope

Back up these authoritative stores:

• PostgreSQL/PostGIS database: tenant data, audit logs, reports, invoices, billing state, sync cursors, route caches, and queue metadata.
• S3-compatible object storage: photos, signatures, generated report PDFs, invoice PDFs, import files, malware-scan results, and thumbnails.
• Secret manager entries: Laravel APP_KEY, database credentials, Redis credentials, S3 credentials, Stripe, Twilio, Expo, Xero, route providers, Reverb, webhook signing secrets, and council transport credentials.
• Deployment configuration: queue workers, scheduler, Horizon, Reverb, storage bucket names, retention settings, and DNS or load-balancer targets.

The runbook is written for PostgreSQL, which is what development uses. The public demo runs MySQL 8: use mysqldump --single-transaction and mysql in place of pg_dump and pg_restore, and the same retention and drill rules apply. See deploy.

Do not treat Redis as authoritative. Redis queues and cache can be rebuilt from database state, scheduled commands, and provider retries.

Recovery Targets

Service level	Database RPO	Object RPO	RTO target
Starter	24 hours	24 hours	8 business hours
Operator	1 hour where provider supports PITR, daily snapshot fallback	1 hour with object versioning	4 business hours
Fleet	15 minutes with WAL/PITR	15 minutes with versioned bucket replication	2 business hours

For production, use managed PostgreSQL continuous backup with point-in-time recovery where available. Keep daily logical dumps as a second recovery path.

Production Backup Policy

PostgreSQL/PostGIS:

• Enable managed continuous backup or WAL archiving.
• Keep daily snapshots for at least 35 days.
• Keep monthly snapshots for at least 13 months.
• Keep logical pg_dump --format=custom --no-owner --no-acl exports daily for at least 14 days. Logical dumps are slower to restore but useful when a provider snapshot is unavailable or corrupt.
• Record backup completion, size, checksum, and provider snapshot ID in the operations log.

Object storage:

• Enable bucket versioning for every production bucket.
• Enable server-side encryption.
• Block public access by default.
• Replicate the bucket to a second region or provider for Fleet tenants.
• Keep deleted object versions for at least 35 days unless tenant contract or legal retention requires more.
• Keep compliance-report and invoice PDF objects for the legal/accounting retention window. Do not lifecycle-delete those objects without a tenant retention rule and legal review.

Secrets:

• Store secrets in a managed secret store, not in git, ticket comments, or local notes.
• Export encrypted secret snapshots after any production secret rotation.
• Keep at least the last two APP_KEY values if key rotation is introduced. Losing the active Laravel key can make encrypted credentials unrecoverable.

Local Backup Commands

Start local infrastructure first:

docker compose -f infra/local-dev/docker-compose.yml up -d

Create a local PostgreSQL custom-format dump:

mkdir -p output/backups/local
backup_file="output/backups/local/postgres-$(date -u +%Y%m%dT%H%M%SZ).dump"

docker compose -f infra/local-dev/docker-compose.yml exec -T postgres \
  pg_dump -U rangefront -d rangefront_waste \
  --format=custom --no-owner --no-acl > "$backup_file"

shasum -a 256 "$backup_file" > "$backup_file.sha256"

If the local S3 bucket is in use, mirror it with an S3-compatible client:

mkdir -p output/backups/local/objects

AWS_ACCESS_KEY_ID=rangefront \
AWS_SECRET_ACCESS_KEY=rangefront-secret \
aws --endpoint-url http://127.0.0.1:9000 \
  s3 sync "s3://${AWS_BUCKET:-rangefront-waste-local}" \
  output/backups/local/objects/

If the bucket does not exist locally, create it before exercising object restore drills:

AWS_ACCESS_KEY_ID=rangefront \
AWS_SECRET_ACCESS_KEY=rangefront-secret \
aws --endpoint-url http://127.0.0.1:9000 \
  s3 mb "s3://${AWS_BUCKET:-rangefront-waste-local}"

Local Restore Drill

Restore to a separate database first. Do not overwrite the working local database until the drill has passed.

restore_file="output/backups/local/postgres-YYYYMMDDTHHMMSSZ.dump"

docker compose -f infra/local-dev/docker-compose.yml exec -T postgres \
  dropdb -U rangefront --if-exists rangefront_waste_restore

docker compose -f infra/local-dev/docker-compose.yml exec -T postgres \
  createdb -U rangefront rangefront_waste_restore

docker compose -f infra/local-dev/docker-compose.yml exec -T postgres \
  pg_restore -U rangefront -d rangefront_waste_restore \
  --no-owner --no-acl --clean --if-exists < "$restore_file"

Point a temporary backend .env at the restore database:

DB_DATABASE=rangefront_waste_restore

Run the app checks:

npm run backend:migrate
npm run backend:test
npm run web:build

Object restore drill:

AWS_ACCESS_KEY_ID=rangefront \
AWS_SECRET_ACCESS_KEY=rangefront-secret \
aws --endpoint-url http://127.0.0.1:9000 \
  s3 sync output/backups/local/objects/ \
  "s3://${AWS_BUCKET:-rangefront-waste-local-restore}"

Then verify a sample compliance report PDF, invoice PDF, and uploaded photo attachment can be resolved from the restored database to object storage.

Production Restore Procedure

1. Declare an incident owner and freeze writes. Put Laravel in maintenance mode, pause schedulers, Horizon workers, Reverb broadcasts, webhook delivery workers, council-report transport workers, and integration sync workers.

2. Choose the restore point. Use the latest safe time before corruption, deletion, or provider outage. Record the snapshot ID, WAL/PITR timestamp, object-storage version marker, and who approved the restore.

3. Restore database to a new instance. Never restore directly over production first. Restore to a new Postgres instance, run migrations only after confirming the restored schema version, and keep the original instance read-only until sign-off.

4. Restore object storage. Restore or expose the matching object versions for attachments, report PDFs, invoice PDFs, and import files. Confirm bucket policies and encryption are still in force.

5. Restore secrets and config. Apply the matching APP_KEY, database credentials, S3 credentials, queue settings, route-provider credentials, Stripe/Twilio/Expo/Xero secrets, and council transport credentials. Do not rotate secrets during the first restore unless the incident is credential compromise.

6. Run integrity checks. Use read-only checks first:

   php artisan migrate:status
   php artisan route:list --path=api/v1
   npm run backend:test

   On production-scale restores, also run SQL spot checks:

   select count(*) from organizations where status = 'active';
   select count(*) from compliance_reports where status in ('issued', 'submitted', 'accepted');
   select count(*) from invoices where status in ('sent', 'partially_paid', 'paid');
   select count(*) from attachments where upload_status = 'ready';
   select count(*) from audit_logs;

7. Verify tenant workflows. For at least one restored tenant:

   • Log in as an owner and a dispatcher.
   • Open customers, properties, onsite systems, jobs, council reports, invoices, settings, and billing.
   • Download one issued compliance report PDF and one invoice PDF.
   • Verify an attachment object path exists and hash matches database metadata.
   • Run a mobile sync pull with a test device.
   • Confirm revoked devices remain blocked.
   • Confirm outbound webhook and council submission queues are paused until operators explicitly resume them.

8. Promote restored environment. Switch application traffic only after the incident owner approves the checklist. Resume schedulers and workers in this order:

   1. Laravel scheduler.
   2. Horizon queues except outbound integrations.
   3. Attachment processing.
   4. Monitoring alerts.
   5. Reverb.
   6. Council and integration transports after checking for duplicate sends.

9. Post-restore cleanup. Keep the old production database and object versions immutable until the incident review is closed. Record data-loss window, failed jobs replayed, tenant notifications sent, and any manual council or accounting corrections.

Monthly Restore Drill

Run a restore drill at least monthly for production and after major schema changes. Store evidence in the operations log:

• Date, operator, source backup ID, target staging environment.
• Database restore duration.
• Object restore duration.
• Schema migration status.
• Sample tenant IDs checked.
• PDF download check result.
• Attachment object/hash check result.
• Mobile sync pull result.
• Queue resume dry-run result.
• Gaps found and owner for each fix.

The drill is not complete until a restored staging environment passes the workflow checks and the evidence is attached to the operations log.

Tenant-Specific Recovery

Use tenant-level recovery only for isolated accidental deletion or import rollback. Tenant recovery must not bypass shared audit history.

• Prefer application-level rollback/replay tools for legacy imports when available.
• For database-level tenant recovery, restore the full backup to staging, export tenant-scoped rows from staging, review foreign keys manually, then import through controlled scripts.
• Never copy one tenant's object prefix into another tenant's prefix.
• Record every manual correction in audit_logs or an incident note tied to the tenant and operator.

Failure Cases

If database restore succeeds but objects are missing:

• Keep the app in maintenance mode for affected tenants.
• Restore object versions or bucket replica first.
• If object loss is partial, identify affected attachments rows and mark customer-visible downloads unavailable until recovered.

If objects restore but database restore fails:

• Keep objects read-only.
• Restore from an older database snapshot, then replay logical dumps or provider WAL where possible.
• Do not reprocess queued outbound webhooks until invoice, report, and council submission state has been checked.

If APP_KEY is missing:

• Stop restore work and escalate.
• Encrypted integration credentials, OAuth tokens, webhook secrets, and billing metadata may be unrecoverable without the correct key.

Owner

The production operator owns this runbook. Update it after every restore drill, provider migration, storage-layout change, queue topology change, or incident.