Claude Code से SaaS और ब्लॉग कमाई के लिए A/B testing

पहले hypothesis लिखें, फिर code

A/B testing सिर्फ दो UI versions दिखाने का तरीका नहीं है। SaaS में यह पूछता है कि pricing CTA, onboarding या trial message से signup और paid intent सच में बेहतर हुए या नहीं। Blog monetization में यह देखता है कि ads, affiliate links, newsletter और consultation CTA से कमाई बढ़ी, लेकिन reading experience, speed और trust खराब तो नहीं हुए। Claude Code toggle code जल्दी लिख सकता है, पर experiment तभी उपयोगी है जब hypothesis, event schema, sample size, guardrail metrics, privacy consent और rollback पहले से तय हों।

सरल भाषा में: variant मतलब test होने वाला version; exposure मतलब user ने कौन सा variant देखा और हमने उसे record किया; guardrail metric मतलब वह safety number जो खराब नहीं होना चाहिए; false positive मतलब random noise को जीत समझ लेना।

Claude Code को business prompt ऐसे दें:

Next.js App Router SaaS/blog के लिए A/B testing workflow बनाएं।
Goal monetization है, vanity clicks नहीं।

experiment id: pricing_page_offer_2026_06
hypothesis: pricing CTA को "Start free trial" से "Start with the free plan" करने पर signup starts बढ़ेंगे और paid-intent clicks नहीं घटेंगे।
primary metric: signup_start_rate
guardrails: purchase_link_click_rate, p75 LCP, JavaScript error rate
deliverables: event schema, server-side assignment, Cookie/localStorage caveats, BigQuery-style SQL, Playwright verification, rollout/rollback checklist.

कम से कम तीन real use cases रखें: SaaS pricing CTA, blog affiliate block की position, newsletter form copy, onboarding step, consultation box. Feature flags के लिए Claude Code feature flags और tracking के लिए Claude Code analytics implementation देखें।

Use case	Primary metric	Guardrails	Common failure
SaaS pricing CTA	Signup start rate	Paid clicks, errors, LCP	Signups बढ़ते हैं पर buyer quality घटती है
Blog affiliate block	Product link click rate	Read completion, bounce, speed	Monetization block बहुत जल्दी आने से trust घटता है
Newsletter form	Completed subscriptions	Spam, unsubscribe	Count बढ़ता है पर list quality घटती है
Onboarding	First success rate	Support tickets, activation	Short-term success later churn छिपाता है

Event schema पहले lock करें

Launch के बाद data mismatch मिलना महंगा है। अगर एक ही click कभी button_click, कभी ctaClicked, कभी signup_click कहलाता है, तो SQL से पहले manual cleanup करना पड़ेगा। Claude Code से UI से पहले typed event contract लिखवाएं। Google Analytics use करते समय official GA4 event reference और Google tag parameter reference देखें।

// lib/experiment-events.ts
export type ExperimentId = "pricing_page_offer_2026_06";
export type VariantId = "control" | "free_plan_copy";

export type ExperimentEvent =
  | {
      event_name: "experiment_exposure";
      experiment_id: ExperimentId;
      variant: VariantId;
      anonymous_id: string;
      page_path: string;
    }
  | {
      event_name: "cta_click";
      experiment_id: ExperimentId;
      variant: VariantId;
      anonymous_id: string;
      cta_id: "pricing_primary" | "article_bottom" | "sidebar_offer";
      page_path: string;
    }
  | {
      event_name: "purchase_link_click";
      experiment_id: ExperimentId;
      variant: VariantId;
      anonymous_id: string;
      product_id: string;
      value_usd: number;
      page_path: string;
    }
  | {
      event_name: "guardrail_metric";
      experiment_id: ExperimentId;
      variant: VariantId;
      anonymous_id: string;
      metric_name: "lcp_ms" | "js_error" | "bounce";
      value: number;
      page_path: string;
    };

declare global {
  interface Window {
    gtag?: (command: "event", name: string, params: Record<string, unknown>) => void;
  }
}

export function trackExperimentEvent(event: ExperimentEvent) {
  if (typeof window === "undefined") return;

  window.gtag?.("event", event.event_name, {
    experiment_id: event.experiment_id,
    variant: event.variant,
    anonymous_id: event.anonymous_id,
    page_path: event.page_path,
    ...event,
  });
}

Events में email, name, company या free-form user text न भेजें। Analytics या advertising storage के लिए consent चाहिए तो tag भेजने से पहले consent state set करें। Google का official consent mode guide यही बताता है।

Server-side assignment करें

सिर्फ localStorage से assignment आसान है, लेकिन first render flicker, login के बाद variant बदलना, private browsing reset और storage blocking जैसे issues देता है। MDN localStorage को origin-based persistent storage बताता है: MDN localStorage। यह first server render के लिए enough source of truth नहीं है।

Next.js App Router में Route Handler अच्छा starting point है। Official route.ts docs custom request handlers समझाते हैं। Cookies के लिए NextResponse देखें। Next.js 16 में Middleware का नाम Proxy हुआ है, इसलिए edge rewrite के लिए proxy.js docs देखें।

// app/api/experiments/assign/route.ts
import { NextRequest, NextResponse } from "next/server";

export const runtime = "edge";

type Variant = "control" | "free_plan_copy";

const EXPERIMENTS = {
  pricing_page_offer_2026_06: {
    cookieName: "ab_pricing_page_offer_2026_06",
    variants: [
      { id: "control", weight: 50 },
      { id: "free_plan_copy", weight: 50 },
    ] satisfies Array<{ id: Variant; weight: number }>,
  },
};

function hashToBucket(input: string) {
  let hash = 2166136261;
  for (let index = 0; index < input.length; index += 1) {
    hash ^= input.charCodeAt(index);
    hash = Math.imul(hash, 16777619);
  }
  return Math.abs(hash) % 100;
}

function chooseVariant(experimentId: keyof typeof EXPERIMENTS, anonymousId: string): Variant {
  const experiment = EXPERIMENTS[experimentId];
  const bucket = hashToBucket(`${experimentId}:${anonymousId}`);
  let cumulative = 0;

  for (const variant of experiment.variants) {
    cumulative += variant.weight;
    if (bucket < cumulative) return variant.id;
  }

  return experiment.variants[0].id;
}

export async function GET(request: NextRequest) {
  const experimentId = request.nextUrl.searchParams.get("experiment");

  if (experimentId !== "pricing_page_offer_2026_06") {
    return NextResponse.json({ error: "Unknown experiment" }, { status: 404 });
  }

  const experiment = EXPERIMENTS[experimentId];
  const testAnonymousId = request.headers.get("x-test-anonymous-id");
  const existingCookie = request.cookies.get(experiment.cookieName)?.value;
  const anonymousId = testAnonymousId ?? existingCookie ?? crypto.randomUUID();
  const variant = chooseVariant(experimentId, anonymousId);

  const response = NextResponse.json({
    experimentId,
    variant,
    anonymousId,
  });

  response.cookies.set(experiment.cookieName, anonymousId, {
    httpOnly: true,
    sameSite: "lax",
    secure: process.env.NODE_ENV === "production",
    path: "/",
    maxAge: 60 * 60 * 24 * 30,
  });

  return response;
}

Cookie भी सावधानी मांगती है। MDN का secure cookie configuration guide Secure, HttpOnly, SameSite समझाता है। Logged-in SaaS hashed user id use कर सकता है; public blog short anonymous cookie रख सकता है; ads measurement में CMP consent पहले आएगा।

Rollout और rollback को config में रखें

Experiment code deploy हो सकता है, लेकिन exposure बिना redeploy बदले। Vercel users Vercel Flags देख सकते हैं; शुरुआत के लिए YAML config पर्याप्त है।

# config/experiments.yaml
experiments:
  pricing_page_offer_2026_06:
    status: running
    owner: masa
    hypothesis: "Free-plan copy increases signup starts without hurting paid intent."
    allocation_percent: 50
    variants:
      control: 50
      free_plan_copy: 50
    primary_metric: signup_start_rate
    guardrails:
      - purchase_link_click_rate
      - p75_lcp_ms
      - js_error_rate
    rollback:
      if_js_error_rate_increases_by: 0.02
      if_p75_lcp_ms_worse_by_ms: 300
      action: "set allocation_percent to 0 and keep logging exposure for audit"

अगर errors बढ़ें, LCP खराब हो, या paid-intent clicks घटें, exposure को 0 करें। अच्छा result दिखे तो भी सीधे 100% न करें; 10%, 50%, फिर 100% rollout करें।

SQL से analyze करें, false positive से बचें

Denominator exposure है। जिसने variant नहीं देखा, उसे count न करें। जिसने multiple variants देखे, उसे exclude या investigate करें। BigQuery में SAFE_DIVIDE zero division से बचाता है।

-- BigQuery Standard SQL
WITH exposure_raw AS (
  SELECT
    anonymous_id,
    experiment_id,
    ARRAY_AGG(variant ORDER BY event_timestamp LIMIT 1)[OFFSET(0)] AS variant,
    MIN(event_timestamp) AS first_exposed_at,
    COUNT(DISTINCT variant) AS variant_count
  FROM `project.dataset.events`
  WHERE event_name = 'experiment_exposure'
    AND experiment_id = 'pricing_page_offer_2026_06'
  GROUP BY anonymous_id, experiment_id
),
exposure AS (
  SELECT anonymous_id, experiment_id, variant, first_exposed_at
  FROM exposure_raw
  WHERE variant_count = 1
),
events_after_exposure AS (
  SELECT
    e.variant,
    e.anonymous_id,
    ev.event_name,
    ev.value_usd,
    ev.value_ms
  FROM exposure e
  LEFT JOIN `project.dataset.events` ev
    ON ev.anonymous_id = e.anonymous_id
   AND ev.experiment_id = e.experiment_id
   AND ev.event_timestamp >= e.first_exposed_at
)
SELECT
  variant,
  COUNT(DISTINCT anonymous_id) AS exposed_users,
  COUNT(DISTINCT IF(event_name = 'cta_click', anonymous_id, NULL)) AS cta_users,
  SAFE_DIVIDE(
    COUNT(DISTINCT IF(event_name = 'cta_click', anonymous_id, NULL)),
    COUNT(DISTINCT anonymous_id)
  ) AS cta_click_rate,
  COUNT(DISTINCT IF(event_name = 'purchase_link_click', anonymous_id, NULL)) AS purchase_intent_users,
  SAFE_DIVIDE(
    COUNT(DISTINCT IF(event_name = 'purchase_link_click', anonymous_id, NULL)),
    COUNT(DISTINCT anonymous_id)
  ) AS purchase_intent_rate,
  AVG(IF(event_name = 'guardrail_metric' AND value_ms IS NOT NULL, value_ms, NULL)) AS avg_guardrail_ms,
  SUM(IF(event_name = 'guardrail_metric' AND value_usd IS NOT NULL, value_usd, 0)) AS revenue_proxy_usd
FROM events_after_exposure
GROUP BY variant
ORDER BY variant;

Sample size launch से पहले तय करें। रोज result देखकर new variant आगे दिखे तो रोक देना false positive बढ़ाता है। बहुत variants, बहुत segments, primary metric बदलना, या paid campaign के साथ test शुरू करना भी risk है।

Playwright से verify करें

Publish से पहले देखें: same anonymous ID को same variant मिले; unknown experiment 404 दे; monetization CTA एक बार render हो। Playwright के official test और expect तथा assertions देखें।

// tests/experiments.spec.ts
import { test, expect } from "@playwright/test";

test.describe("pricing_page_offer_2026_06", () => {
  test("keeps assignment stable for the same anonymous id", async ({ request, baseURL }) => {
    const url = `${baseURL}/api/experiments/assign?experiment=pricing_page_offer_2026_06`;
    const headers = { "x-test-anonymous-id": "demo-user-42" };

    const first = await request.get(url, { headers });
    const second = await request.get(url, { headers });

    expect(first.ok()).toBeTruthy();
    expect(second.ok()).toBeTruthy();
    expect(await first.json()).toMatchObject(await second.json());
  });

  test("rejects unknown experiments", async ({ request, baseURL }) => {
    const response = await request.get(`${baseURL}/api/experiments/assign?experiment=missing`);

    expect(response.status()).toBe(404);
  });

  test("renders one monetization CTA on the pricing page", async ({ page }) => {
    await page.goto("/pricing?e2e_anonymous_id=demo-user-42");

    await expect(page.getByTestId("pricing-cta")).toBeVisible();
    await expect(page.getByTestId("pricing-cta")).toHaveCount(1);
  });
});

Masa ने article CTA checks में पाया कि सबसे ज्यादा फायदा SQL से नहीं, पहले event table लिखने से हुआ। कौन सा click monetization CTA है, exposure कब record होगा, और multiple variants वाले users का क्या होगा, यह साफ होने से Claude Code पर rework कम हुआ। एक छोटे test में Playwright ने localStorage flicker पकड़ा और implementation server cookie पर वापस गया।

Real funnel में लागू करना हो तो Claude Code training और product templates देखें। लक्ष्य ज्यादा experiments नहीं, बल्कि safe experiments हैं जो revenue question का साफ answer दें।