# Disaster Recovery Runbook — RankWizAI

**Maintained by:** solo operator  
**Last reviewed:** 2026-06-07  
**RTO target:** 4 hours (from incident declaration to app serving traffic)  
**RPO target:** 24 hours (daily backup at 02:00 UTC; worst case = yesterday's data)

---

## Recovery Scenarios

### Scenario A: Database Corruption / Accidental Deletion

**Symptoms:** Application errors referencing missing tables; data visible in UI yesterday but gone today.

**Steps:**

1. **Put app in maintenance mode immediately to prevent further writes:**
   ```bash
   php artisan down --retry=60 --refresh=30
   ```

2. **Identify the latest good backup:**
   ```bash
   # List backups on the VPS (local storage)
   ls -lt storage/app/backups/ | head -10

   # Or list from S3 (requires AWS CLI configured)
   aws s3 ls s3://${AWS_BACKUP_BUCKET}/backups/ --human-readable | sort -k1,2 -r | head -10
   ```

3. **Download the latest backup from S3 if local copy is unavailable:**
   ```bash
   BACKUP_FILE="backups/rankwiz-YYYY-MM-DD-HHMMSS.sql.gz"
   aws s3 cp "s3://${AWS_BACKUP_BUCKET}/${BACKUP_FILE}" /tmp/restore-backup.sql.gz
   ```

4. **Verify integrity before restoring:**
   ```bash
   # Check gzip magic bytes
   xxd /tmp/restore-backup.sql.gz | head -1
   # Expected: 00000000: 1f8b ...

   # Decompress and inspect header
   gzip -dc /tmp/restore-backup.sql.gz | head -5
   # Expected: -- MySQL dump ... or -- MariaDB dump ...

   # Or run the built-in drill (never touches production):
   php artisan backup:drill --show-files
   ```

5. **Restore the database (MySQL):**
   ```bash
   # Create a safety snapshot of current (possibly corrupt) DB first
   mysqldump -u ${DB_USERNAME} -p${DB_PASSWORD} ${DB_DATABASE} | gzip > /tmp/pre-restore-snapshot.sql.gz

   # Drop and recreate the database
   mysql -u ${DB_USERNAME} -p${DB_PASSWORD} -e "DROP DATABASE ${DB_DATABASE}; CREATE DATABASE ${DB_DATABASE} CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"

   # Restore from backup
   gzip -dc /tmp/restore-backup.sql.gz | mysql -u ${DB_USERNAME} -p${DB_PASSWORD} ${DB_DATABASE}
   echo "Restore exit code: $?"
   ```

6. **Clear Laravel caches after restore:**
   ```bash
   php artisan cache:clear
   php artisan config:cache
   php artisan route:cache
   php artisan view:cache
   ```

7. **Bring the app back online:**
   ```bash
   php artisan up
   php artisan queue:restart
   ```

8. **Verify by checking key pages in the browser and monitoring logs:**
   ```bash
   tail -f storage/logs/laravel.log
   ```

---

### Scenario B: VPS Hardware Failure / Single-Server Total Loss

**Symptoms:** SSH unreachable; hosting provider reports hardware failure; entire VPS gone.

**Steps:**

1. **Provision a new VPS** (same provider or alternative). Minimum spec: 2 vCPU, 4 GB RAM, 50 GB SSD.

2. **Install dependencies on the new server:**
   ```bash
   # Ubuntu/Debian:
   apt update && apt install -y nginx php8.4-fpm php8.4-mysql php8.4-redis php8.4-gd php8.4-curl php8.4-zip php8.4-bcmath php8.4-mbstring php8.4-xml git unzip mysql-client redis-server supervisor nodejs npm jq
   ```

3. **Clone the repository:**
   ```bash
   git clone git@github.com:YOUR_ORG/rankwiz.git /var/www/rankwiz
   cd /var/www/rankwiz
   ```

4. **Restore `.env` from a secure secret store (1Password / Bitwarden / SSM Parameter Store):**
   ```bash
   # Copy your production .env from secret store
   # All required variables are documented in .env.example
   ```

5. **Install PHP and JS dependencies:**
   ```bash
   composer install --no-dev --optimize-autoloader
   npm ci --prefer-offline
   npm run build
   ```

6. **Restore the database from S3** (follow Scenario A steps 2–5).

7. **Run pending migrations (in case the backup predates recent migrations):**
   ```bash
   php artisan migrate --force
   ```

8. **Configure nginx + PHP-FPM** using your existing nginx config from the repository (`/docs/server-config/nginx.conf` if present) or the standard Laravel vhost template.

9. **Configure Supervisor** for queue workers:
   ```bash
   # Install supervisor config
   cp /var/www/rankwiz/docs/server-config/supervisor-horizon.conf /etc/supervisor/conf.d/
   supervisorctl reread && supervisorctl update && supervisorctl start all
   ```

10. **Schedule the cron job:**
    ```bash
    crontab -e
    # Add:
    * * * * * cd /var/www/rankwiz && php artisan schedule:run >> /dev/null 2>&1
    ```

11. **Point DNS to the new server IP** (TTL permitting; use low TTL in advance).

12. **Verify:** run `php artisan backup:drill --show-files` and check `/up` returns HTTP 200.

---

### Scenario C: Deploy Failure with Partial Migration

**Symptoms:** `deploy.sh` fails mid-run; app may be in maintenance mode; `artisan migrate:status` shows partial migration.

**Action:** `deploy.sh` automatically calls `migrate:rollback --step=N` in its `rollback()` trap (DEPLOY-002). Verify the rollback succeeded:

```bash
php artisan migrate:status
# Confirm the migrations that ran during the failed deploy show as "Pending"

php artisan up
# Bring the app back to the previous working state
```

If the automatic rollback failed (logged as a warning by `deploy.sh`):
```bash
# Manual rollback — check how many were applied in the failed deploy
php artisan migrate:rollback --step=1  # repeat for each applied migration
php artisan up
```

---

## Maintenance Window Duration (DEPLOY-003 / RISK-003)

The current deploy strategy is `DEPLOY_STRATEGY=maintenance_window`. Expected downtime:

| Phase | Duration |
|-------|----------|
| Composer install (pre-downtime) | ~20-40s |
| npm build (pre-downtime) | ~30-60s |
| **artisan down** (window starts) | instant |
| migrate --force | **~1-10s** (additive migrations only) |
| cache rebuild | ~2s |
| **artisan up** (window ends) | instant |
| Total user-visible 503 window | **~10-15s** |

Long-term path to zero-downtime: symlink-based releases (blue/green). Accepted risk for current
solo-operator deployment volume. Revisit when traffic warrants (<10 active users currently).

---

## S3 Lifecycle Policy

Apply to `${AWS_BACKUP_BUCKET}` via AWS Console or CLI:

```bash
aws s3api put-bucket-lifecycle-configuration \
  --bucket ${AWS_BACKUP_BUCKET} \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "backup-glacier-transition",
        "Status": "Enabled",
        "Filter": {"Prefix": "backups/"},
        "Transitions": [{"Days": 90, "StorageClass": "GLACIER_IR"}]
      },
      {
        "ID": "backup-expiry",
        "Status": "Enabled",
        "Filter": {"Prefix": "backups/"},
        "Expiration": {"Days": 365}
      }
    ]
  }'
```

Config reference: `config/backup.php` `s3_lifecycle` keys (editable via env `BACKUP_S3_GLACIER_DAYS`, `BACKUP_S3_EXPIRY_DAYS`).

---

## Monthly Drill Verification

The `backup:drill` command runs automatically on the 1st of each month at 05:00 UTC. 
To run manually: `php artisan backup:drill --show-files`

A failed drill logs `CRITICAL` and fires `FailedBackupNotification` (see `BACKUP_ALERT_EMAIL`). 
**A failed drill does NOT mean data is lost** — it means the verification step failed and you should manually inspect the latest S3 backup.

---

*For other runbooks (GSC token revoked, Stripe webhook failure, queue backlog), see the other files in `docs/runbooks/`.*