# Runbook: Queue Backlog

**Category:** Infrastructure · Queue  
**Frequency:** Occasionally during traffic spikes or worker outages  
**Severity:** P2 — jobs delayed (analysis, drafts, sync); P1 if `failed_jobs` is growing (data loss risk)

---

## Trigger

- Horizon shows queued jobs older than 5 minutes on the `default` or `gsc-sync` queue
- `failed_jobs` table count is growing faster than jobs are being retried
- Users report "analysis is stuck", "drafts never generated", or "sync not running"
- PagerDuty alert: `HorizonInactiveAlert` or `QueueSizeAlert`

---

## Symptoms

| Signal | Where to look |
|--------|--------------|
| Horizon queue size > N (site-dependent baseline) | `/horizon` admin panel |
| `failed_jobs.created_at` advancing quickly | DB query below |
| `dead_letter_jobs` table growing | DB query + admin DLQ UI |
| Analysis runs stuck in `processing` status for >10 min | DB query below |
| Workers not visible in Horizon → Processes | Horizon → Monitoring |

**Quick DB checks:**
```sql
-- Jobs stuck in processing (analysis runs not completing)
SELECT ar.id, s.domain, ar.status, ar.started_at, TIMESTAMPDIFF(MINUTE, ar.started_at, NOW()) AS min_running
FROM analysis_runs ar
JOIN sites s ON s.id = ar.site_id
WHERE ar.status = 'processing'
  AND ar.started_at < NOW() - INTERVAL 10 MINUTE
ORDER BY ar.started_at;

-- Failed jobs snapshot
SELECT queue, COUNT(*) AS count, MAX(failed_at) AS latest
FROM failed_jobs
GROUP BY queue
ORDER BY count DESC;

-- Dead letter queue snapshot
SELECT job_class, COUNT(*) AS count, MAX(created_at) AS latest, SUM(retryable::int) AS retryable_count
FROM dead_letter_jobs
GROUP BY job_class
ORDER BY count DESC
LIMIT 20;
```

---

## Diagnosis

### Step 1 — Check Horizon health

Open the Horizon dashboard at `/horizon` (admin only):
- **Workers panel** — are worker processes present? If empty, workers are down.
- **Queues panel** — look at throughput (jobs/min). A high queue size with zero throughput = dead workers.
- **Failed jobs panel** — see which job classes are failing.

### Step 2 — Identify the backlog type

**Type A — Workers stopped** (queue size growing, zero throughput):
→ Jump to Resolution, Section A.

**Type B — Burst traffic** (workers running, queue growing slowly):
→ The queue will drain on its own. Monitor for 15 minutes. If not trending down, scale workers (Section B).

**Type C — Specific job class looping into failures** (one job class dominates `failed_jobs`):
→ That job class has a bug or its dependency is down. Jump to Section C.

**Type D — Jobs completing but stuck in `processing` state** (analysis_runs):
→ Worker process killed mid-job; the run status was never flipped to `completed` or `failed`. Jump to Section D.

### Step 3 — Check worker process health

```bash
# On the worker host — supervisor status
sudo supervisorctl status

# Laravel queue:work process check
ps aux | grep "queue:work" | grep -v grep
```

---

## Resolution

### Section A — Workers stopped: restart

```bash
# Restart supervisor (which restarts queue:work processes)
sudo supervisorctl restart rankwiz-worker:*

# Signal a graceful restart (tells in-flight jobs to finish, then respawn)
php artisan queue:restart

# Verify workers came back up
php artisan horizon:status
```

Monitor Horizon for 2–3 minutes post-restart. Throughput should climb back to baseline within 1 minute of workers spawning.

### Section B — Burst: scale workers temporarily

If a legitimate spike (e.g. a batch AI job, mass site imports):
```bash
# Spawn additional queue:work processes manually (temporary, not persisted)
nohup php artisan queue:work --queue=default --sleep=3 --tries=3 --max-time=3600 &

# When the backlog clears, terminate the extra processes
kill <pid>
```

For persistent scaling, update `config/horizon.php` `environments.production.supervisor-1.numProcesses` and redeploy.

### Section C — Specific job class failing repeatedly

1. Identify the failing class in `failed_jobs.payload` (job_class field).
2. Check the most recent exception:
   ```bash
   php artisan tinker --execute="
     DB::table('failed_jobs')
       ->latest('failed_at')
       ->limit(5)
       ->get(['id', 'queue', 'exception'])
       ->each(fn(\$j) => print \$j->exception . PHP_EOL . '---' . PHP_EOL);
   "
   ```
3. If the dependency is down (e.g. OpenAI API, GSC API): do NOT retry immediately. Let the backoff schedule run. Set a timer to re-check in 30 minutes.
4. If it's a code bug (not an external dependency): retry is futile until code is deployed. Move jobs to DLQ:
   - Open admin DLQ UI at `/admin/dlq`
   - Filter by job class
   - Mark non-retryable + bulk-delete, or wait for a fix deploy then bulk-replay

### Section D — Stuck `processing` analysis runs

The worker process was killed mid-job. The run will never self-resolve.

```bash
php artisan tinker --execute="
  // Reset to 'failed' so the user can re-trigger
  \App\Models\AnalysisRun::where('status', 'processing')
    ->where('started_at', '<', now()->subMinutes(15))
    ->update([
      'status' => 'failed',
      'error' => 'Job process killed — worker restarted. Please run the analysis again.',
      'completed_at' => now(),
    ]);
  print 'Reset complete.';
"
```

After reset, notify affected users if support has open tickets for them.

### Section E — DLQ triage via admin UI

The admin DLQ at `/admin/dlq` lets you:
- **Preview** a failed job's payload (sensitive fields masked automatically)
- **Replay** a single job (re-queues it; marks `replayed_at`)
- **Bulk replay** all retryable jobs
- **Bulk delete** jobs that are non-retryable or whose fix has been deployed
- **Purge** jobs older than N days (housekeeping — safe to purge after 30 days)

Only use bulk replay when you've fixed the underlying cause. Replaying without fixing the bug just refills `failed_jobs`.

---

## Escalation

| Condition | Action |
|-----------|--------|
| All workers down and supervisor won't restart | SSH to host, check `systemctl status supervisor`; restart host if supervisor is wedged |
| DLQ growing faster than you can triage | Disable the job class dispatch at the controller level (feature flag or code deploy), then triage from a stopped state |
| `failed_jobs` > 10,000 rows | Table scan performance may degrade. Purge old entries first: `php artisan queue:flush` (clears ALL failed_jobs — confirm before running in production) |
| Memory leak suspected in workers | Check `queue:work --max-jobs=500` flag — limit per-process job count so workers recycle |

---

## Related

- Admin DLQ UI: `/admin/dlq` (admin only)
- Horizon dashboard: `/horizon` (admin only)
- `app/Http/Controllers/Admin/AdminDlqController.php` — DLQ CRUD + bulk replay/purge
- `app/Services/Queue/DeadLetterService.php` — DLQ service (replay logic, retryability check)
- `app/Jobs/` — all queue job classes; `$tries` and `$backoff` are set per-class
- `config/horizon.php` — worker count, queue configuration