# Implementation Milestones — RankWizAI Test Harness *Companion to `e2e-testing-proposal.md`. The proposal is strategic; this is operational.* This document breaks the v2 plan into executable milestones with explicit deliverables, acceptance criteria, dependencies, risks, and de-risk strategies. The final section (**Gap analysis**) lists items that were missing or under-specified in the v2 proposal and where they have been incorporated. Conventions: - **M-IDs** are stable references. M0 through M10 are the spine. - **Effort** is in focused hours, not calendar time. Calendar windows assume ~10–15 hrs/week alongside product work. - **Acceptance criteria** are the binary check that "this milestone is done." Anything not in acceptance criteria is scope creep. - **De-risk** describes what to do if the milestone runs over budget or hits a blocker. The honest answer is sometimes "punt the milestone." --- ## Milestone overview and dependency graph ``` M0 (pre-flight) ↓ M1 (static analysis) ──────────────────┐ ↓ │ M2 (scenario engine) ──┐ │ ↓ │ │ M3 (WP integration) ─┐ │ │ │ │ │ M4 (fakes) ───┴──┴──→ M5 (scenario coverage) ↓ M6 (drift canary) ──┐ M7 (mutation tests) ┤ ↓ │ M8 (hygiene rituals + docs) ↓ M9 (deferred: prod-to-fixture bridge) ↓ M10 (post-implementation review) ``` M3, M4, M6, M7 are partly parallelizable. M5 is the synthesis milestone; it cannot start until M2/M3/M4 are at least at MVP. | Milestone | Calendar window | Effort | Critical path? | |---|---|---|---| | M0 — Pre-flight & inventory | Week 0 | 4–6 hrs | Yes | | M1 — Static analysis hardening | Weeks 1–2 | 24–32 hrs | Yes | | M2 — Scenario engine MVP | Weeks 3–4 | 28–36 hrs | Yes | | M3 — WordPress integration | Weeks 5–6 | 24–32 hrs | Partial | | M4 — External fakes | Weeks 5–7 (parallel with M3 tail) | 28–36 hrs | Partial | | M5 — Scenario coverage to all rules | Weeks 7–8 | 24–32 hrs | Yes | | M6 — Drift canary | Weeks 9–10 | 14–18 hrs | No | | M7 — Mutation testing | Weeks 9–10 (parallel with M6) | 24–32 hrs | No | | M8 — Hygiene rituals & documentation | Weeks 11–12 | 12–16 hrs | Yes | | M9 — Prod-to-fixture bridge (deferred) | TBD | 8–12 hrs | No | | M10 — Post-implementation review | Week 13 | 4 hrs | Yes | **Total: 194–266 hours over 13 calendar weeks.** The high end of the range is the realistic plan; the low end assumes nothing goes sideways. v2's "150–180 hours" estimate was already optimistic; this is the corrected number. --- ## M0 — Pre-flight & inventory **Goal:** Establish the baseline. Know what exists, what's missing, what's broken, and what the constraints are before writing a line of new code. **Deliverables:** - Coverage baseline report: `./vendor/bin/pest --parallel --coverage --coverage-html=storage/coverage` run once; HTML output committed snapshot or saved. - A `decision-log.md` file in `docs/qa-strategy/` capturing every "we chose X over Y because Z" call (Dusk vs. Playwright, SQLite vs. MySQL for tests, etc.). Living document. - Confirmation that Docker Desktop or equivalent is installed on dev machine (M3 dependency). - An honest assessment of which existing tests are AI-generated and likely-confident-but-wrong; flag for later sampling review. - A list of every external service the codebase touches (read from `config/services.php` + `bootstrap/providers.php` + grep for HTTP clients). This is the canonical "things that need fakes" list. - CI runner spec confirmed: GitHub Actions runner OS, PHP version, Node version, MySQL version, available services. Document in `decision-log.md`. **Acceptance criteria:** - [ ] Coverage report exists and the current overall % is recorded - [ ] `decision-log.md` exists with at least 5 decisions captured - [ ] Docker is installed and `docker --version` succeeds - [ ] External service inventory committed - [ ] CI environment characteristics documented **Dependencies:** None. **Effort:** 4–6 hrs. **Risks:** - Coverage tooling not configured. *De-risk:* skip the report; record overall test count instead. - Docker not feasible on dev machine. *De-risk:* push M3 to use a remote test container (e.g., a permanent dockerized WP at a known URL). --- ## M1 — Static analysis hardening (Layer 0) **Goal:** Make a large class of AI-generated bugs unmergeable at static-analysis time. **Deliverables:** PHPStan/Larastan: - PHPStan pushed to level 9 (or as close as your Eloquent annotations allow). Resulting errors fixed. - A custom Larastan rule: `LazyLoadingViolationRule` — flags `$model->relationship` access in code paths that don't have a preceding `->load()` or `->loadMissing()`. Targets Gotcha #1. - A custom Larastan rule: `WpApiClientReturnTypeRule` — flags `WpApiClient::getContent()` results being passed where a string is expected. Targets Gotcha #6. - A custom Larastan rule: `JobTriesPropertyRule` — flags `protected $retries = ...` on classes implementing `ShouldQueue` and suggests `$tries`. Targets Gotcha #12. - A custom Larastan rule: `BannedMiddlewareRule` — flags any reference to `AddLinkHeadersForPreloadedAssets`. Targets Gotcha #15. - A custom Larastan rule: `InertiaPageExistsRule` — flags `Inertia::render('X/Y', ...)` where `resources/js/Pages/X/Y.tsx` does not exist. Targets Gotcha #10. Frontend static checks: - TypeScript `strict` enabled in `tsconfig.json` (verify it already is; tighten if not). - ESLint custom rule: flag `dangerouslySetInnerHTML` without a `sanitizeHtml` import in the same file. Targets Gotcha #8. - ESLint config: ban `any` (already in place; verify enforcement). CI gates: - A custom `pest --filter=ContractTests/ServiceHasTest` test or PHP CLI script that walks `app/Services/`, asserts each service has a corresponding test file. Fails CI if violated. - A `pest --filter=ContractTests/MigrationHasTest` test for any new migration that lacks a feature test asserting behavior (lower priority; phase 2). - A CI step that runs `php artisan ziggy:generate --check` (or equivalent) to fail if Ziggy is out of sync. Targets Gotcha #13. - Pre-commit hook: husky + lint-staged running `pint`, `eslint`, `tsc --noEmit`, `pest --dirty`, and PHPStan on changed files only. **Acceptance criteria:** - [ ] `vendor/bin/phpstan analyse --memory-limit=1G` passes at level 9 (or documented level with reason) - [ ] All five custom Larastan rules implemented, with at least one passing/failing test each - [ ] CI fails when a service is added without a test (verified with a test PR) - [ ] CI fails when Ziggy is out of sync (verified with a test PR) - [ ] Pre-commit hook is installed and runs in <30s on a typical changeset **Dependencies:** M0. **Effort:** 24–32 hrs. (PHPStan upgrade alone may absorb 10+ hrs depending on existing violations.) **Risks:** - Level 9 surfaces hundreds of errors. *De-risk:* settle for level 8; commit the upgrade path as a tracked tech debt item. - Larastan custom rules require deeper PHPStan AST knowledge than expected. *De-risk:* implement the simplest two rules (banned middleware, banned `$retries`) as grep-based checks instead of AST rules; defer the hard ones. - Pre-commit hook is annoyingly slow. *De-risk:* gate behind a `RANKWIZ_FAST_COMMIT=1` env var for emergency commits. **Notes:** This milestone is the highest-ROI work in the entire plan. Do not skip rules to save time — every rule is worth more than several scenario tests because it runs on every line, on every commit, exhaustively. --- ## M2 — Scenario engine MVP **Goal:** Establish the pattern. One contract, three scenarios, one Dusk test green in CI. Everything downstream depends on this shape. **Deliverables:** Contract & registry: - A `SeededState` value object (immutable; `User`, `Site`, `Vec`, `array $extra` for scenario-specific data, plus a `loginUrl` or auth token). - A `Scenario` interface: `seed(): SeededState`. - A `ScenarioRegistry` that maps scenario class names to instances. - A Pest helper `scenario(string $name): SeededState` that resolves and seeds. - A Dusk helper `loginAsScenarioUser(SeededState $state): Browser` that handles auth bridging. Three scenarios: - `SiteWithStrikingDistanceQueries` — 50 pages, 30 with positions 11–20, hand-crafted impressions. - `SiteWithCannibalCluster` — 5 pages competing for 1 keyword cluster, with overlap thresholds chosen to trigger `CannibalizationDetectionService`. - `SiteAtPlanLimit` — user one draft below `drafts_per_month` limit; subscription state seeded via Cashier directly. Dusk integration: - Dusk installed, `php artisan dusk:install`, `.env.dusk.testing` configured for SQLite. - One Dusk test: `tests/Browser/RunsAnalysisOnStrikingDistanceTest.php` — logs in, visits site dashboard, runs analysis, asserts a striking-distance recommendation appears. - Dusk runs in CI (GitHub Actions) on PRs. Async / job handling: - Pest test trait `RunsScenarioJobs` that calls `Queue::fake()` and provides helpers to run jobs synchronously when needed. Document the pattern in `tests/CLAUDE.md`. - Decision recorded: scenarios that need to test job-graph behavior end-to-end run jobs synchronously via `Queue::fake()->assertPushed()` + manual dispatch; scenarios that don't, ignore the queue. Avoid integration tests that wait on real queues. Authorization scaffolding: - Every scenario seeds a user that owns the site. A second helper, `SeededState::asNonOwner(): User`, returns a different user for authorization tests. Tests that exercise `SitePolicy` denial paths use this. Migration testing scaffold: - A Pest contract test `tests/Feature/MigrationsRunCleanlyTest` that runs `migrate:fresh --seed` and asserts no errors. Catches AI-broken migrations. - A Pest contract test that asserts every `Site` has a non-null `gsc_property_url` after seeding (sanity check; expand the pattern as needed). **Acceptance criteria:** - [ ] Three scenario classes exist, each with a passing seed test (calls `->seed()` and asserts the returned state structurally) - [ ] One Dusk journey test passes in local CI and on GitHub Actions - [ ] `tests/CLAUDE.md` updated with the scenario pattern + job-fake pattern - [ ] Migration contract test passes - [ ] Scenario authorization helper tested with one denial-path test **Dependencies:** M0, M1 (uses the service-test-existence check). **Effort:** 28–36 hrs. **Risks:** - Dusk in GitHub Actions is fiddly (Chrome version mismatches, headless flakiness). *De-risk:* use a known-good action (e.g., `nanasess/setup-chromedriver`) and pin versions. Document the working config aggressively. - Scenario seed time blows out as scenarios accumulate. *De-risk:* benchmark seed time per scenario; if any exceeds 500ms, optimize before adding more. - Job-graph testing pattern is unclear. *De-risk:* commit to `Queue::fake()` + manual dispatch as the canonical pattern; do not try to test "real queue ordering" until M5. --- ## M3 — WordPress integration test suite **Goal:** A small, dedicated test suite that exercises the Laravel↔WP plugin seam against a real containerized WordPress, including HMAC handshake and bidirectional publishing. **Deliverables:** Container setup: - `docker-compose.test.yml` defining: `wordpress` (stable WP image, version pinned, e.g., `wordpress:6.4-php8.1-apache`), `wp-mysql` (MariaDB), and a volume mount for `wp-plugin/rankwiz-ai/`. - A `bin/test-wp-up.sh` script that brings up the stack, waits for WP to be ready, runs `wp-cli` to install WP, activate the plugin, create test users, and seed test posts. - A `bin/test-wp-down.sh` script for teardown. - WP HMAC shared secret synchronized between Laravel test config and WP plugin via env injection at container boot. Test infrastructure: - A `RefreshesAllDatabases` test trait that resets both Laravel's DB (via `RefreshDatabase`) and the WP container's DB (via `wp-cli db reset --yes` + re-seed). - A `WpIntegrationTestCase` base class that includes the trait and sets the HTTP client base URL to the container. - A `tests/Browser/WpIntegration/` directory exclusively for these tests; runs sequentially, marked with `@group wp-integration`. - CI step: a separate GitHub Actions job that runs only `pest --group=wp-integration` against the docker-compose stack. Concrete tests: - HMAC handshake test: Laravel calls WP's discovery endpoint; signature validated; response received. - Content sync test: Laravel calls WP's content listing endpoint; `WpApiClient::getContent()` array unwrap verified. - Bidirectional publish test: Laravel publishes a draft → WP creates a post → assert via WP REST API that the post exists with correct content. - Webhook receipt test: trigger a WP webhook to Laravel's `WpWebhookController`; assert the corresponding `ProcessWpWebhookJob` ran with the right payload. WP plugin internal testing (gap fill): - A separate, lighter test suite *inside* `wp-plugin/rankwiz-ai/tests/` using PHPUnit (WP plugins don't easily run Pest). Tests the plugin's REST routes, HMAC verification, and content extraction in isolation. Run on every PR that touches the plugin directory. WP version drift handling: - A second CI job, gated to nightly only: same WP integration suite against `wordpress:latest` and `wordpress:6.7-php8.1-apache` (or current N and N-1). Catches "we work on 6.4 but break on 6.7" issues. Failures are warnings, not blockers. **Acceptance criteria:** - [ ] `docker-compose -f docker-compose.test.yml up` brings the stack to ready state in <60s - [ ] Four WP integration tests pass locally and in CI - [ ] WP plugin internal test suite has at least 5 tests covering HMAC + content endpoints - [ ] Nightly cross-version job exists and runs (failures as warnings) - [ ] `RefreshesAllDatabases` trait verified to reset both DBs cleanly between tests **Dependencies:** M0 (Docker), M2 (scenario pattern for the Laravel side). **Effort:** 24–32 hrs. **Risks:** - Container startup is too slow for CI. *De-risk:* keep the integration suite small (<20 tests), run it as a single sequential job, accept 3–5 minute total runtime. - WP-CLI seeding is fragile. *De-risk:* invest in idempotent seed scripts; rerun on failure. - The plugin's internal test suite was never a thing before. *De-risk:* start with three tests; expand only if bugs are caught there. - Webhook receipt test requires a tunnel from WP → Laravel. *De-risk:* in test, both run on the same Docker network; use `host.docker.internal` or the network alias directly. --- ## M4 — External fakes (OpenAI, DataForSEO, Stripe, OAuth) **Goal:** Hermetic, deterministic, failure-mode-injectable fakes for every non-WP external service. **Deliverables:** OpenAI fake: - `tests/Fakes/FakeOpenAi.php` — implements your `OpenAiService` contract or wraps the underlying client. - Prompt-fingerprint matching: hash of normalized prompt → fixture file in `tests/fixtures/openai/{fingerprint}.json`. - Recording mode: a `RECORD_OPENAI_FIXTURES=1` env var that, when set in a special test run, captures live responses and writes new fixture files. (Requires real API key; gate carefully.) - Failure-mode helpers: `FakeOpenAi::failNext('rate_limit')`, `'timeout'`, `'malformed_json'`, `'refusal'`, `'empty'`, `'oversized'`. Tests opt in. - Strict mode: an unknown prompt fingerprint throws `FixtureMissingException` loudly. No silent fallback. DataForSEO fake: - Same shape as OpenAI fake. `tests/fixtures/dataforseo/{endpoint}/{fingerprint}.json`. - Failure-mode helpers: `'rate_limit'`, `'task_failed'`, `'empty_serp'`, `'auth_error'`. Stripe fake (using Cashier + Test Mode): - A `SeedsStripeSubscription` trait that uses Cashier model methods to create subscription state directly. No HTTP calls. - A `tests/fixtures/stripe-webhooks/` directory with captured webhook payloads (one per event type your app handles) generated via `stripe trigger`. - A `dispatchStripeWebhook(string $event, array $overrides = [])` helper that POSTs the fixture to your webhook endpoint with a valid signature. OAuth fake (Google for GSC): - A standalone fake OAuth provider service (small Express app or PHP-based test server) that responds to `/authorize` and `/token` endpoints with fixed behavior. - A test trait `UsesFakeOAuth` that points the GSC client base URL at the fake during tests. - One end-to-end OAuth flow test exercising the full handshake. GSC HTTP-level fake (sync pipeline only): - For tests that exercise `GscSyncService` end-to-end, `Http::fake()` with a small library of canned responses in `tests/fixtures/gsc/`. - For analysis tests, scenarios seed `GscDailyMetric/PageMetric/QueryMetric` directly. Document the distinction. Email/notification fake (gap fill): - Default to Laravel's `Mail::fake()` and `Notification::fake()` in scenario tests. Document the assertion pattern in `tests/CLAUDE.md`. - Snapshot tests for the rendered HTML of every transactional email (welcome, draft published, ROI alert, traffic anomaly). Snapshots in `tests/snapshots/emails/`. Encrypted column round-trip (gap fill): - One Pest test asserting that `encrypted` casts work end-to-end: write encrypted → read decrypted → assert equality, against `User::ai_key`, `Site::wp_secret`, etc. Caching / limit invalidation (gap fill): - One Pest test per `LimitCacheService::invalidateCount()` consumer asserting that an Observer change does invalidate the relevant cache key. **Acceptance criteria:** - [ ] Each fake has at least one happy-path scenario test using it - [ ] Each fake has at least one failure-mode test (e.g., `'rate_limit'`) - [ ] Recording mode for OpenAI works on demand (don't run it in CI) - [ ] OAuth happy-path E2E passes - [ ] Email snapshot tests exist for every transactional email type - [ ] Encrypted-column round-trip test passes - [ ] At least one cache-invalidation test passes per cache consumer **Dependencies:** M2 (test scaffolding). **Effort:** 28–36 hrs. **Risks:** - Prompt fingerprint matching is brittle (whitespace changes break it). *De-risk:* normalize aggressively before hashing — strip trailing whitespace, collapse runs, lowercase. - Stripe webhook fixtures drift when Stripe updates their event format. *De-risk:* the drift canary covers this (M6). - Fake OAuth provider is fiddly. *De-risk:* use `stevenmaguire/oauth2-fake-server` or similar pre-built solution if available; otherwise hand-roll a 50-line PHP app. --- ## M5 — Scenario coverage to all rules, detectors, limits **Goal:** Every recommendation rule, every opportunity detector, and every plan-limit boundary has at least one scenario fixture and one journey test. **Deliverables:** Recommendation rule scenarios (one per rule): - `SiteWithNoindexedHighTrafficPage` → triggers `NoindexRule` - `SiteWithThinContent` → triggers `ThinContentRule` - `SiteWithStaleContent` → triggers `FreshnessRecommendationRule` - `SiteWithMissingMetaTags` → triggers `MetaTagRule` - `SiteNeedingInternalLinks` → triggers `InternalLinkingRule` - `SiteNeedingContentRewrite` → triggers `ContentRewriteRule` - `SiteNeedingEeatSignals` → triggers `EeatRecommendationRule` - `SiteNeedingGeoOptimization` → triggers `GeoRecommendationRule` - (Plus the cannibal cluster scenario from M2) Opportunity detector scenarios: - (Striking distance from M2) - `SiteWithCtrGap` → triggers `KeywordOpportunityService::detectCtrGaps()` - `SiteWithRisingQueries` → triggers `detectRisingQueries()` - `SiteWithContentGap` → triggers `detectContentGaps()` Limit-boundary scenarios: - (Plan limit from M2) - `UserAtSitesLimit` - `UserAtSyncFrequencyLimit` - `UserAtSeatsLimit` - `UserOverLimitFromOverride` — exercises the per-user/per-site override cascade Async / job-graph scenarios (gap fill): - `WebhookArrivalReorderingScenario` — exercises out-of-order WP webhook handling - `JobRetryAfterFailureScenario` — exercises `$tries` + backoff behavior on a failing job - `RoiCalculationAfterSyncScenario` — exercises the ROI baseline → sync → recalculation chain Authorization edge-case scenarios (gap fill): - `CrossTenantAccessAttemptScenario` — user A tries to access user B's site; SitePolicy denies - `ExpiredSubscriptionAccessScenario` — subscription canceled mid-session; feature-gated routes return correct response Audit log scenario (gap fill): - `AuditLogIntegrityScenario` — exercises a write to the audit log that uses only `action`, `site_id`, `context`, `ip_address`. Targets Gotcha #3. Soft-delete scenarios (gap fill): - One scenario per soft-deleted model (`Site`, `User`, `BlogPost`, `WebhookEndpoint`, `ChangelogEntry`) that asserts deleted records are excluded from default queries but included via `withTrashed()`. Feature-flag scenarios (gap fill): - Tests that exercise both states of every flag in `config/features.php`. Use Pest's `dataset` to parameterize. Scheduled-command coverage (gap fill): - Each command in `php artisan schedule:list` has at least one Pest test invoking the underlying command and asserting expected side effects (DB changes, jobs queued, notifications sent). Journey tests: - For each scenario above, at least one Pest feature test (controller/job-level) calling the underlying analysis or detection code and asserting the recommendation/opportunity/denial appears. - For the top 5 user journeys (chosen by revenue impact / complaint rate), a Dusk browser test exercising the journey end-to-end. **Acceptance criteria:** - [ ] One scenario per recommendation rule (8 total + cannibal = 9) - [ ] One scenario per opportunity detector (4 total) - [ ] One scenario per limit boundary (5 total) - [ ] Async/job-graph scenarios exist (3 total) - [ ] Authorization, audit log, soft-delete, feature flag, scheduled command coverage in place - [ ] Top 5 Dusk journey tests pass in CI - [ ] Total test runtime stays under 15 min for the full PR-gate suite **Dependencies:** M2, M3, M4. **Effort:** 24–32 hrs. **Risks:** - Test runtime balloons. *De-risk:* parallelize Pest aggressively; move expensive scenarios to a `nightly` group; profile and optimize the slowest 10%. - Scenario duplication temptation grows. *De-risk:* if you find yourself writing similar `seed()` methods, extract a `BaseSiteScenario` parent class. But not before. - Some rules require GSC distributional shapes that are hard to hand-craft. *De-risk:* introduce a tiny helper `GscRowFactory::position($n)->impressions($n)->ctr($n)` to make hand-crafting less painful, but resist building a "realistic distribution" engine. --- ## M6 — Drift canary **Goal:** Nightly insurance against upstream API changes. Failure is loud. **Deliverables:** Test accounts: - A real GSC property under your control with at least 90 days of history. (Probably already exists.) - A real WordPress test site at a known URL with the plugin installed. ($5/mo VPS or shared host.) - DataForSEO sandbox account with a small monthly budget cap. - OpenAI sandbox key with a low spend cap (~$10/mo). - Stripe Test Mode (free). Canary suite (`tests/Canary/`): - `GscCanaryTest` — performs a real GSC sync of the test property; asserts response shape against pinned schema. - `WpCanaryTest` — performs a real HMAC handshake + content fetch against the test site. - `OpenAiCanaryTest` — sends one fixed prompt; asserts response shape parses. - `DataForSeoCanaryTest` — runs one fixed query; asserts response shape parses. - `StripeCanaryTest` — creates a test subscription via Cashier; asserts state transitions correctly via webhook. Schema pinning: - Every external API response has a Zod-equivalent schema (use `spatie/laravel-data` with strict mode, or hand-write JSON Schema and validate via `opis/json-schema`). - Schemas live in `tests/Canary/Schemas/`. - When a schema mismatch occurs, the failure includes a structured diff (new fields added, fields removed, type changes) — not just "test failed." Token / secret management: - A `.env.canary` file holding canary credentials, encrypted at rest, decrypted only in the canary CI job. - A token refresh helper for GSC OAuth — runs before the canary; if the token is within 7 days of expiry, attempts refresh; if it fails, alerts loudly. Alerting: - Two consecutive nights of failure pages you (Sentry alert, email, Slack — pick the channel that hurts most). - A single night of failure logs to a dashboard but doesn't page (avoid alert fatigue). - A schema diff is included in the alert payload. Cron / scheduling: - GitHub Actions scheduled workflow at 03:00 UTC daily. - Run completes in <10 minutes total. - Results posted to a `canary-history.md` log committed nightly (last 30 days retained). **Acceptance criteria:** - [ ] All five canary tests pass against real APIs at least once (manual run) - [ ] Scheduled workflow runs in CI and completes successfully on a known-good day - [ ] Alert wired and tested (force-fail one canary; confirm page received on second night) - [ ] Schema diff format verified by hand-mutating a schema - [ ] Token refresh helper tested on a near-expiry token **Dependencies:** M4 (fakes establish the schemas; canary asserts real responses match them). **Effort:** 14–18 hrs. **Risks:** - GSC OAuth refresh fails silently. *De-risk:* the refresh helper logs aggressively; an alert fires on refresh failure separately from data failure. - DataForSEO or OpenAI changes pricing model and the canary becomes expensive. *De-risk:* monthly cost cap on accounts; if exceeded, canary is auto-disabled and you get an email. - Test WP site goes down (hosting issue). *De-risk:* the canary distinguishes "site unreachable" from "site responded with wrong shape"; only the latter is treated as drift. --- ## M7 — Mutation testing onboarding **Goal:** A meaningful mutation score floor on critical business logic. The AI-code-quality gate, made real. **Deliverables:** Setup: - Infection installed via Composer, configured in `infection.json5`. - Targets restricted to a curated path list: `app/Services/RecommendationEngine.php`, `app/Services/Rules/*`, `app/Services/RoiCalculationService.php`, `app/Services/LimitService.php`, `app/Services/PlanLimitService.php`. *Not* the full `app/`. - A baseline run completed; results triaged. Triage: - All surviving mutants categorized: real bugs (write a test), equivalent mutants (annotate `@infection-ignore-all`), edge cases worth deferring (note in `infection-debt.md`). - A score floor set at the post-triage value. Likely 60–75% initially. CI integration: - Nightly CI job runs the full mutation suite against the curated paths. - A separate PR-gate job runs Infection only on files changed in the PR (`infection --git-diff-base=main`). Fails the PR if the score on changed files drops below floor. - Score history committed to `infection-history.md` for trend visibility. Discipline: - A `infection-debt.md` document tracks the equivalent mutants and edge cases. Reviewed monthly during hygiene week. - New code in mutation-targeted paths is expected to maintain or improve the score. **Acceptance criteria:** - [ ] Infection runs cleanly on the curated paths (no infrastructure errors) - [ ] Score floor documented in `infection.json5` `minMsi` setting - [ ] At least 10 surviving mutants triaged and either killed or annotated - [ ] PR-gate job passes on a no-op PR - [ ] PR-gate job fails on a deliberately-broken PR (verify with a test PR) **Dependencies:** M2 (good tests are a prerequisite for meaningful mutation testing). **Effort:** 24–32 hrs. *Most of this is triage, not setup.* **Risks:** - The first run reports 30% score and a wall of mutants. *De-risk:* this is normal; budget the triage time, don't panic. - Infection is slow (>30 min). *De-risk:* curate the path list aggressively; nightly is fine for slow runs. - Equivalent mutants pile up. *De-risk:* maintain `infection-debt.md` with discipline; hygiene-week reviews keep it bounded. --- ## M8 — Hygiene rituals & documentation **Goal:** The harness is operational, sustainable, and documented. A new contributor (or future-you, or AI-with-context) can use it without tribal knowledge. **Deliverables:** Documentation: - `docs/qa-strategy/README.md` — one-page tour: layers, where things live, how to add a scenario, how to add a test, how to handle a canary alert, how to add a fake. - `docs/qa-strategy/playbooks/` directory with short playbooks: "When canary fails on shape diff," "When mutation score drops on PR," "When a new external service is added," "When a new recommendation rule is added." - `tests/CLAUDE.md` updated to reference the harness comprehensively. AI-context discipline. - A "context bundle" CLI: `php artisan ai-context Service\\RecommendationEngine` prints the relevant CLAUDE.md sections, gotchas, existing tests, and interfaces. Used to seed AI prompts. Weekly hygiene ritual: - A calendar block (Friday 60 min) covering: sample-based test-of-tests review (5 random tests, 30 min), one adversarial scenario script run (30 min). - A weekly checklist committed to the repo: `docs/qa-strategy/weekly-hygiene-checklist.md`. Adversarial scenario tooling: - `php artisan scenarios:adversarial RuleName` — calls Claude/GPT with the rule source + existing scenarios + a structured prompt requesting boundary scenarios. Outputs scenario class skeletons to `storage/adversarial/`. Human triage produces final scenarios. Onboarding (gap fill): - A `bin/setup-test-env.sh` script that bootstraps the entire test stack on a fresh clone: installs deps, builds Docker images, pulls fixture files, runs migrations. Documented in the top-level README. - A 5-minute "first contribution" walkthrough doc. Decision log finalization: - `decision-log.md` is consolidated and archived. Major decisions captured: framework choices, fake strategies, mutation targets, canary scope. **Acceptance criteria:** - [ ] `docs/qa-strategy/README.md` covers all eight layers/milestones - [ ] At least four playbooks written - [ ] `php artisan ai-context X` works on three sample inputs - [ ] Weekly hygiene checklist exists and was followed for two consecutive weeks - [ ] `bin/setup-test-env.sh` runs cleanly on a fresh clone (verify in a sandbox) - [ ] Decision log committed **Dependencies:** All prior milestones. **Effort:** 12–16 hrs. **Risks:** - Documentation drifts as soon as it's written. *De-risk:* link CLAUDE.md → playbooks → tests bidirectionally; treat doc updates as part of any test infrastructure change. - The weekly hygiene ritual gets skipped. *De-risk:* calendar it as immovable for the first month; honest assessment after that whether to keep it. --- ## M9 — Production-to-fixture bridge (deferred) **Goal:** When a production exception occurs, generate a draft scenario stub. Optional milestone; only build if value is demonstrated. **Deliverables:** - A Sentry webhook receiver in your Laravel app that, on certain exception classes, generates a `storage/scenario-candidates/{timestamp}-{exception}.php` file containing: stack trace, request context, relevant model snapshots (sanitized), and a scenario class skeleton. - A weekly review of `scenario-candidates/` during hygiene block. - A simple promotion path: a candidate becomes a real scenario when human triage confirms it. **Acceptance criteria:** - [ ] Webhook generates a candidate file for a deliberately-thrown exception - [ ] At least one candidate has been promoted to a real scenario - [ ] Privacy/PII scrubbing in the snapshot logic verified **Dependencies:** M5 (need the scenario pattern). **Effort:** 8–12 hrs. **Decision criterion:** Build M9 only if, by the end of M8, you have observed at least 3 production-only bugs that *would have* been caught by the scenario pattern but weren't. Otherwise the ROI isn't there. --- ## M10 — Post-implementation review **Goal:** Honest assessment of whether the harness is paying off. Refactor scope; cut what isn't earning its keep. **Deliverables:** - Coverage report comparison: M0 baseline vs. post-M8 state. - Bug capture stats: how many bugs did each layer catch in the last 13 weeks? (Static analysis, scenario tests, mutation, canary, hygiene review.) - Time investment vs. estimate: actual hours per milestone vs. budget. - A "kill list" of things that aren't paying off — defunct scenarios, low-value tests, brittle fakes. - Recommendations for next quarter. **Acceptance criteria:** - [ ] All metrics gathered and documented - [ ] At least one explicit kill (a test or a piece of infrastructure removed) - [ ] Plan for next quarter committed **Dependencies:** All prior. **Effort:** 4 hrs. --- ## Cross-cutting concerns (gap fill) Items that don't belong to a single milestone but must be threaded through several. Each is owned by the milestone in parentheses. - **Job queue testing pattern** (M2): Established. `Queue::fake()` + manual dispatch. Used throughout M5. - **Authorization (SitePolicy) testing** (M2 trait, M5 scenarios): Cross-tenant denial paths covered. - **Audit log integrity** (M5): Explicit scenario covers the column-drop gotcha. - **Soft-delete behavior** (M5): Per-model scenarios. - **Feature flag matrix** (M5): Datasets covering both states of each flag. - **Scheduled command coverage** (M5): One test per command. - **Email/notification fakes** (M4): `Mail::fake()` + `Notification::fake()` pattern + snapshot tests. - **Encrypted column round-trip** (M4): One test. - **Cache invalidation correctness** (M4): One test per `LimitCacheService` consumer. - **Migration cleanliness** (M2): `MigrationsRunCleanlyTest`. - **Ziggy sync** (M1): CI check. - **Performance / N+1** (M1+M2): Lazy-loading violations are caught at runtime by Laravel's strict mode (already enabled per CLAUDE.md) plus the Larastan custom rule. - **Security / XSS** (M1): ESLint rule for `dangerouslySetInnerHTML`. - **WP plugin internal tests** (M3): Separate suite. - **WP version drift** (M3): Nightly cross-version job. - **Two-database isolation** (M3): `RefreshesAllDatabases` trait. - **OAuth refresh in canary** (M6): Refresh helper. - **Secret management for canary** (M6): `.env.canary` encrypted at rest. --- ## Risk register Risks that span milestones, ordered by impact × likelihood. | # | Risk | Impact | Likelihood | Mitigation | |---|---|---|---|---| | R1 | AI generates the test infrastructure too, with the same plausible-but-wrong issues. | High | High | Sampling-based test-of-tests in M8. Pair-write the foundational pieces (M1 rules, M2 contract) without AI assistance, or with extremely tight specs. | | R2 | Time estimates are still optimistic; rollout takes 18 weeks instead of 13. | Medium | High | Acknowledge upfront. M5 is the natural compression target — defer scenarios to phase 2 if behind. | | R3 | Dusk is flaky in CI. | Medium | Medium | Pin Chrome version aggressively. Retry transient failures up to 2x. If flake rate >5%, switch to Playwright (eat the cost). | | R4 | WP container startup pushes total CI runtime past 20 min. | Medium | Medium | Keep WP suite tightly scoped (<20 tests). Run only on PRs that touch WP-relevant paths. | | R5 | Mutation testing triage is so painful you abandon it. | Medium | Medium | Start with 1 file, not 10. Expand only when triage habit forms. Floor at post-triage value, not aspirational value. | | R6 | Drift canary alerts become noise; you stop reading them. | High | Low | Two-night rule (single failure logs only; two consecutive nights pages). Schema-diff in alerts to make triage 2 minutes. | | R7 | A new external service is added during rollout and breaks the model. | Medium | Low | The "external service inventory" in M0 is a living doc. New services get a fake before merge. | | R8 | The canary's test domains/accounts get suspended (DataForSEO billing, GSC ownership change). | Medium | Low | Document the recovery procedure in M8 playbooks. | | R9 | Production-only bugs continue at high rate despite harness. | High | Low | Trigger M9 (production-to-fixture bridge) earlier than planned. | | R10 | A milestone unblocks slower than expected (e.g., M3 takes 50 hrs instead of 28). | Medium | Medium | Each milestone has explicit de-risk strategies. M3 specifically can be reduced to "dockerized WP + HMAC test only" if budget runs out, deferring publish/webhook tests. | --- ## Gap analysis: what was missing or under-specified in v2 These are gaps I caught while writing the milestone breakdown — items the v2 proposal under-specified or missed entirely. Each is now incorporated into a specific milestone. 1. **Migration testing.** v2 didn't mention it. AI-generated migrations are dangerous (column drops, foreign-key changes, irreversible operations). Added to M2 as `MigrationsRunCleanlyTest`. 2. **Job queue testing pattern.** v2 said "scenarios exercise the full job graph" but never specified *how*. Added to M2: `Queue::fake()` + manual dispatch as the canonical pattern, documented in `tests/CLAUDE.md`. 3. **Authorization (SitePolicy) testing.** v2 mentioned scenarios but didn't address that site-scoped authorization is a critical security boundary. Added to M2 (trait helper) and M5 (cross-tenant denial scenarios). 4. **Audit log integrity.** v2 listed Gotcha #3 but didn't add a scenario. Added to M5. 5. **Soft-delete behavior.** v2 didn't mention it. Five models in your codebase use it (`Site`, `User`, `BlogPost`, `WebhookEndpoint`, `ChangelogEntry`). Added to M5. 6. **Feature flag testing.** v2 didn't mention. Added to M5 with parameterized datasets. 7. **Scheduled command coverage.** v2 didn't address `php artisan schedule:list`. Added to M5. 8. **Email and notification testing.** v2 didn't mention. Added to M4 with `Mail::fake()` + snapshot tests for transactional emails. 9. **Encrypted column round-trip.** v2 didn't mention the BYOK encrypted-cast columns. Added to M4 as a one-test sanity check. 10. **Cache invalidation correctness.** v2 didn't address the limit-cache invalidation chain. Added to M4. 11. **WordPress version drift.** v2 didn't address that customers run different WP versions. Added to M3 as a nightly cross-version job. 12. **Two-database test isolation.** v2 mentioned this in the adversarial review but didn't operationalize it. Now a concrete deliverable in M3 (`RefreshesAllDatabases` trait). 13. **WP plugin internal tests.** v2 omitted entirely. The plugin is a separate codebase with its own concerns. Added to M3. 14. **Ziggy sync as a CI check.** v2 listed Gotcha #13 but didn't add a CI gate. Added to M1. 15. **Pre-commit hook installation.** v2 mentioned it conceptually; M1 makes it a concrete deliverable. 16. **Secret management for the canary.** v2 said "real test accounts" but didn't address how OAuth tokens, API keys, and credentials are stored. Added to M6 (`.env.canary` encrypted at rest). 17. **Token refresh handling.** v2 mentioned OAuth tokens expire but didn't specify how the canary handles it. Added to M6. 18. **Onboarding script.** v2 didn't address fresh-clone setup. Added to M8 (`bin/setup-test-env.sh`). 19. **AI-context CLI.** v2 mentioned a "pre-prompt context bundle" but didn't operationalize. Added to M8 (`php artisan ai-context X`). 20. **Decision log.** v2 didn't propose tracking decisions. Added to M0 and finalized in M8. 21. **Post-implementation review.** v2 didn't include a "did this work?" checkpoint. Added as M10. 22. **Schema diff in canary alerts.** v2 said "shape assertion" but didn't specify the operator-friendly diff format. Added to M6 as a concrete deliverable. 23. **CI runner specification.** v2 didn't address what CI environment is assumed. Added to M0. 24. **Async / job-graph scenarios.** v2 mentioned async ordering as a threat but didn't add scenarios for it. Added to M5 (`WebhookArrivalReorderingScenario`, `JobRetryAfterFailureScenario`, `RoiCalculationAfterSyncScenario`). 25. **Recording mode for OpenAI fixtures.** v2 said "recorded fixtures" but didn't address how they're recorded. Added to M4 (`RECORD_OPENAI_FIXTURES=1` env mode). 26. **Strict mode for unknown prompt fingerprints.** v2 said fakes are "deterministic" but didn't address the silent-fallback failure mode. Added to M4 as `FixtureMissingException`. 27. **Cross-tenant access scenario.** v2 mentioned authorization but didn't specify the scenario. Added to M5. 28. **GSC HTTP-level fakes vs. database-level seeding.** v2 said "scenarios seed the database" but didn't address that *sync* tests need HTTP-level fakes. Added to M4 with explicit guidance on which path to use when. These are the gaps. The plan as written now covers them. There may be more — gaps are infinite — but these are the ones I would have most regretted shipping without. --- ## A final honest note Implementation plans are aspirations dressed up as commitments. This one is more honest than v1, but it is still a plan. Things will go wrong. Milestones will run over. Some scenarios will turn out to be the wrong abstraction. The drift canary will alert at 4am on something benign. A WP update will break the integration suite the day before a release. M9 will probably never get built. The plan's job is not to predict reality. Its job is to give you a known-good sequence to fall back on when reality gets confusing, and to make the gaps in your thinking visible early enough to fix. If the milestones survive contact with the first three weeks of execution roughly intact, this document did its job. If they don't, revise the document — that's also doing its job. Build the harness boringly. Update the plan honestly. Ship.