# AIO Recovery — Custom Metrics Emission Guide (W2-C)

This document describes how each custom Datadog metric referenced in
`aio_recovery_dashboard.json` and `aio_recovery_alerts.yaml` is produced.

All thresholds are configured in `config/aio_recovery_monitoring.php`.

---

## Metric Emission Methods

### A. Log-Based Metric Rules (Datadog Logs → Metrics)

These are derived from structured `Log::info('aio_recovery.run.completed', [...])` output
emitted by `AioRecoveryDetector::detect()` (W2-C addition).

Create the following **Datadog Log-Based Metric** rules in the Datadog UI or Terraform:

| Metric Name | Log Query Filter | Value | Group By |
|---|---|---|---|
| `rankwiz.aio_recovery.pages_flagged` | `source:php message:"aio_recovery.run.completed"` | `@pages_flagged` | `site_id`, `env` |
| `rankwiz.aio_recovery.items_by_bucket` (tag: `bucket:high`) | same | `@bucket_high` | `site_id`, `env` |
| `rankwiz.aio_recovery.items_by_bucket` (tag: `bucket:medium`) | same | `@bucket_medium` | `site_id`, `env` |
| `rankwiz.aio_recovery.items_by_bucket` (tag: `bucket:low`) | same | `@bucket_low` | `site_id`, `env` |
| `rankwiz.aio_recovery.est_clicks_recoverable` | same | `@est_clicks_recoverable` | `site_id`, `env` |

### B. Job Status Counters (from Laravel Horizon / queue metrics)

These counters come from Horizon's built-in metrics or a lightweight Artisan command
that pushes counts to DogStatsd. Until a dedicated emitter is built, use the
Datadog log-based approach against Horizon's own logs:

| Metric Name | Source | Notes |
|---|---|---|
| `rankwiz.aio_recovery.runs_completed` | Log query on `message:"Analysis run completed"` + `run_type:AioRecoveryRun` | Count per minute |
| `rankwiz.aio_recovery.runs_failed` | Log query on `message:"Analysis run failed"` + `run_type:AioRecoveryRun` | Count per minute |
| `rankwiz.aio_recovery.job_failure_rate` | Derived: `runs_failed / (runs_failed + runs_completed)` | Computed in Datadog formula widget |

### C. Queue Depth (Horizon Metrics)

`rankwiz.queue.background_depth` — emit via a scheduled Artisan command or
read from Horizon's Redis keys:

```php
// In a scheduled command (e.g., App\Console\Commands\EmitQueueMetricsCommand):
$depth = app(\Illuminate\Queue\QueueManager::class)->connection()->size('background');
// Push via DogStatsd: \Datadog\DogStatsd::gauge('rankwiz.queue.background_depth', $depth, ['env:production']);
```

Horizon already tracks this internally; the simplest path is a Horizon webhook or
the Horizon metrics API endpoint.

### D. SERP Spend (from platform_serp_spend table)

`rankwiz.serp.platform_spend_mtd_usd` — emit via a scheduled Artisan command:

```php
// Run every 15 minutes (schedule::everyFifteenMinutes())
$month = now()->format('Y-m');
$spend = \App\Models\PlatformSerpSpend::where('month', $month)->value('usd_spent') ?? 0.0;
// Push: \Datadog\DogStatsd::gauge('rankwiz.serp.platform_spend_mtd_usd', (float) $spend, ['env:production']);
```

### E. OpenAI / Bundled AI Cost (from MonitorBundledAiCostJob)

`rankwiz.ai.bundled_cost_daily_usd` — `MonitorBundledAiCostJob` already runs hourly and
logs the `daily_cost_usd` field. Add a log-based metric rule:

| Metric Name | Log Query Filter | Value |
|---|---|---|
| `rankwiz.ai.bundled_cost_daily_usd` | `message:"bundled_ai_cost.hourly_check"` | `@daily_cost_usd` |

### F. ROI Tiles (v1-ready, populate after Wave 4)

These emit after Wave 4 ships:

| Metric Name | Source |
|---|---|
| `rankwiz.aio_recovery.clicks_recovered_mtd` | `analytics.aio_recovery_realized` event + `clicks_delta` prop |
| `rankwiz.aio_recovery.items_active` | Count of `aio_recovery_items` with `status=active` via scheduled command |
| `rankwiz.analytics.aio_recovery_*` | Log-based metrics from `analytics.*` audit log events |

---

## Staging Validation Checklist

Before GA, confirm each alert in `aio_recovery_alerts.yaml` trips under synthetic load:

- [ ] **Detection volume anomaly** — create a run with `items_created = 0` (cliff), confirm alert fires within 10 min.
- [ ] **Job failure rate** — force-fail 3 of 5 `RunAioRecoveryJob` dispatches via `queue:fail`; confirm 60% rate triggers critical.
- [ ] **SERP spend warn** — seed `platform_serp_spend.usd_spent = 21.00` for current month; confirm warn alert fires.
- [ ] **SERP spend critical** — seed `usd_spent = 24.00`; confirm critical alert fires.
- [ ] **OpenAI spend** — seed `ai_drafts` rows such that `MonitorBundledAiCostJob.calculateCosts()` returns `daily_cost > 25`; confirm critical fires.
- [ ] **Queue backpressure warn** — push 201 synthetic jobs to `background` queue; confirm warn fires.
- [ ] **Queue backpressure critical** — push 501 jobs; confirm critical fires.

All alerts must fire within their `evaluation_delay` window. Document results in the
Wave 5-F review report.