1. Definition

  • The true conversion rate (p) is the actual probability that a user (or customer) in the entire population will convert (e.g., click, purchase, sign up).
  • It’s a population parameter → fixed but usually unknown.
  • What we measure in an experiment is the sample conversion rate ($\hat{p}$​), which is just an estimate of the true conversion rate.

2. Formula (Population Level)

$p = \frac{\text{Number of Conversions in Population}}{\text{Total Users in Population}}$

  • Since we usually cannot observe the entire population, we estimate with:

$\hat{p} = \frac{x}{n}$

Where:

  • $x$ = number of conversions in the sample
  • $n$ = total number of users in the sample
  • $\hat{p}$​ = sample conversion rate (estimator of true p)

3. Example

Case 1 – Sample (Experiment)

  • 1,000 users saw version A, and 50 converted.

$\hat{p}_A = \frac{50}{1000} = 0.05 \; (5\%)$

Case 2 – True Conversion Rate

  • If we could measure all users in the world exposed to version A, maybe the actual proportion is p = 0.052 (5.2%).
  • But we don’t know this exactly → we only estimate it with $\hat{p}$​.

4. Confidence Interval for True Conversion Rate

Since the sample estimate has uncertainty, we build a CI to capture the true rate:

$CI = \hat{p} \pm Z_{\alpha/2} \cdot \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

Example:

  • $\hat{p} = 0.05$, $n = 1000$, α = 0.05 (95% confidence).
  • SE = $\sqrt{\frac{0.05 \times 0.95}{1000}} \approx 0.0069$
  • CI = $0.05 \pm 1.96 \times 0.0069 \approx [0.036, 0.064]$.

Interpretation: We are 95% confident the true conversion rate lies between 3.6% and 6.4%.

5. Why It Matters in A/B Testing

  • We never know the true conversion rate.
  • We estimate it with a sample and compare across groups (control vs treatment).
  • Hypothesis tests (e.g., two-proportion z-test) help decide if differences between estimated rates are statistically significant → evidence about differences in the true conversion rates.

6. Key Takeaways

  • True conversion rate (p): Fixed but unknown proportion of users who would convert in the full population.
  • Sample conversion rate ($\hat{p}$​): Observed conversions in your experiment → random estimate of p.
  • Confidence intervals & hypothesis testing let us infer the likely value of the true conversion rate.

In short:
The true conversion rate is the actual proportion of users who would convert in the entire population. Since we can’t measure everyone, we use the sample conversion rate ($\hat{p}$​) from experiments to estimate it, and apply confidence intervals and tests to infer its likely value.