How to run A/B tests using feature flags without extra tools

A/B testing doesn’t always require expensive experimentation platforms or complex vendor integrations. If your development workflow already uses feature flags, you’re one step away from running meaningful experiments without adding new tools or infrastructure overhead.

Modern feature flag systems handle user segmentation, variant assignment, and exposure tracking out of the box. By repurposing these capabilities for controlled experiments, teams can validate product changes faster, reduce operational complexity, and avoid the steep licensing fees that come with specialized A/B testing software.

From Feature Flag to A/B Test in Minutes

The core of A/B testing with feature flags lies in reorienting how you use an existing capability. A standard feature flag splits users into two groups: those who see a new feature and those who don’t. An A/B test does the same thing, but with a clear objective — to measure which version performs better against a predefined metric.

const showNewPricing = rollgate.isEnabled('new-pricing-page', { userId });

if (showNewPricing) {
  renderNewPricingPage();  // Variant B
  track('pricing_page_view', { variant: 'new' });
} else {
  renderCurrentPricingPage();  // Variant A (control)
  track('pricing_page_view', { variant: 'control' });
}

The code remains unchanged. The difference is intentional: you’re not just toggling a feature on or off — you’re measuring outcomes and making decisions based on data, not assumptions.

Ensuring Reliable User Assignment

Consistency is non-negotiable in A/B testing. A user must see the same variant every time they interact with the experiment, regardless of device, browser, or session. Cookie-based assignment fails here because users switch devices or clear cookies, breaking experiment integrity.

Quality feature flag systems use deterministic hashing on stable identifiers like userId. The system hashes the identifier using algorithms such as MurmurHash, producing a consistent value between 0 and 100. If your rollout is set to 50%, users whose hash falls below 50 see the new variant; others see the control.

const variant = rollgate.isEnabled('new-pricing-page', {
  userId: user.id,  // Stable identifier ensures consistency
});

This server-side approach works across web, mobile, API, and even email campaigns — no cookies, no device dependency, and no manual targeting rules.

Planning Your Experiment: Start with a Hypothesis

Before writing code or setting flags, define what success looks like. A vague goal like "Let’s test the new pricing page" leads to unclear results. Instead, craft a testable hypothesis:

• Bad: "We should update the pricing page CTA."
• Good: "Changing the CTA from ‘Start Free Trial’ to ‘Get Started Free’ will increase trial signups by at least 15% within two weeks."

With a clear hypothesis, you know exactly what to measure and when to conclude the experiment.

Choosing the Right Metric and Sample Size

Selecting the wrong metric is a common pitfall. Teams often track too many primary metrics, inflating the risk of false positives — finding a statistically significant result that isn’t real.

Focus on one primary metric that directly reflects business impact. Secondary metrics can provide context, but only one should determine success or failure.

| Experiment Type | Good Primary Metric | Avoid Tracking | |-------------------------------|----------------------------|------------------------------------| | Pricing page redesign | Trial signups | Page views, time on page | | Checkout flow optimization | Completed purchases | Cart additions, page views | | Search algorithm update | Click-through on first result | Total searches, session duration |

Sample size is equally critical. Running an experiment for two days with 100 users won’t yield reliable insights. Use statistical power calculations to determine how many users you need per variant.

A practical rule of thumb:

• At a 5% conversion rate baseline, you need roughly 1,500 users per variant to detect a 20% relative improvement.
• At a 2% baseline, aim for 4,000 users per variant.
• High-traffic pages may reach sufficient data in days.
• Low-traffic pages might require weeks or longer.

Resist the urge to stop early. Premature conclusions are as unreliable as flipping a coin five times and declaring it biased after four heads.

Putting It All Together: A Step-by-Step Implementation

Here’s how to run a pricing page A/B test using feature flags in a production environment.

Step 1: Define the Flag and Rollout

Create a boolean feature flag named experiment-pricing-cta with a 50% rollout targeted at all logged-in users. This ensures balanced exposure while maintaining experiment integrity.

Step 2: Instrument the Code

// Server-side (Node.js)
app.get('/pricing', async (req, res) => {
  const showNewCTA = rollgate.isEnabled('experiment-pricing-cta', {
    userId: req.user.id,
  });

  analytics.track('experiment_exposure', {
    experiment: 'pricing-cta',
    variant: showNewCTA ? 'new-cta' : 'control',
    userId: req.user.id,
  });

  res.render('pricing', { showNewCTA });
});

// Client-side (React)
function PricingPage() {
  const showNewCTA = useFlag('experiment-pricing-cta');

  useEffect(() => {
    analytics.track('experiment_exposure', {
      experiment: 'pricing-cta',
      variant: showNewCTA ? 'new-cta' : 'control',
    });
  }, [showNewCTA]);

  return (
    <div>
      <h1>Choose your plan</h1>
      <Button onClick={handleSignup}>
        {showNewCTA ? 'Get Started Free' : 'Start Free Trial'}
      </Button>
    </div>
  );
}

Step 3: Track Key Actions

When a user completes the target action — in this case, signing up for a trial — log the conversion with the variant they experienced.

function handleSignup(plan) {
  analytics.track('trial_signup', {
    experiment: 'pricing-cta',
    variant: rollgate.isEnabled('experiment-pricing-cta', { userId }) ? 'new-cta' : 'control',
    plan: plan,
  });
}

Step 4: Analyze and Act on Results

After reaching your target sample size, export the data and compute conversion rates per variant. Use statistical tests like chi-square or t-tests to validate significance.

SELECT 
  variant,
  COUNT(DISTINCT user_id) AS users,
  COUNT(DISTINCT CASE WHEN converted THEN user_id END) AS conversions,
  ROUND(
    COUNT(DISTINCT CASE WHEN converted THEN user_id END)::numeric /
    COUNT(DISTINCT user_id) * 100,
    2
  ) AS conversion_rate
FROM experiment_events
WHERE experiment = 'pricing-cta'
GROUP BY variant;

If the new variant outperforms the control by a statistically significant margin, roll it out permanently. Otherwise, iterate or abandon the change based on evidence.

The Future of Experimentation: Simpler, Smarter, Faster

Feature flags are no longer just for toggling features on and off — they’re a powerful engine for data-driven product development. By integrating experimentation into your existing workflow, you eliminate vendor sprawl, reduce costs, and accelerate decision-making.

The next time you consider adding another tool to your stack, ask whether your feature flag system can do the job. With the right setup, it probably can.