True Conversion Rate

1. Definition

• The true conversion rate (p) is the actual probability that a user (or customer) in the entire population will convert (e.g., click, purchase, sign up).
• It’s a population parameter → fixed but usually unknown.
• What we measure in an experiment is the sample conversion rate ($\hat{p}$​), which is just an estimate of the true conversion rate.

---

2. Formula (Population Level)

$p = \frac{\text{Number of Conversions in Population}}{\text{Total Users in Population}}$

• Since we usually cannot observe the entire population, we estimate with:

$\hat{p} = \frac{x}{n}$

Where:

• $x$ = number of conversions in the sample
• $n$ = total number of users in the sample
• $\hat{p}$​ = sample conversion rate (estimator of true p)

---

3. Example

Case 1 – Sample (Experiment)

• 1,000 users saw version A, and 50 converted.

$\hat{p}_A = \frac{50}{1000} = 0.05 \; (5\%)$

Case 2 – True Conversion Rate

• If we could measure all users in the world exposed to version A, maybe the actual proportion is p = 0.052 (5.2%).
• But we don’t know this exactly → we only estimate it with $\hat{p}$​.

---

4. Confidence Interval for True Conversion Rate

Since the sample estimate has uncertainty, we build a CI to capture the true rate:

$CI = \hat{p} \pm Z_{\alpha/2} \cdot \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$

Example:

• $\hat{p} = 0.05$, $n = 1000$, α = 0.05 (95% confidence).
• SE = $\sqrt{\frac{0.05 \times 0.95}{1000}} \approx 0.0069$
• CI = $0.05 \pm 1.96 \times 0.0069 \approx [0.036, 0.064]$.

Interpretation: We are 95% confident the true conversion rate lies between 3.6% and 6.4%.

---

5. Why It Matters in A/B Testing

• We never know the true conversion rate.
• We estimate it with a sample and compare across groups (control vs treatment).
• Hypothesis tests (e.g., two-proportion z-test) help decide if differences between estimated rates are statistically significant → evidence about differences in the true conversion rates.

---

6. Key Takeaways

• True conversion rate (p): Fixed but unknown proportion of users who would convert in the full population.
• Sample conversion rate ($\hat{p}$​): Observed conversions in your experiment → random estimate of p.
• Confidence intervals & hypothesis testing let us infer the likely value of the true conversion rate.

---

In short:
The true conversion rate is the actual proportion of users who would convert in the entire population. Since we can’t measure everyone, we use the sample conversion rate ($\hat{p}$​) from experiments to estimate it, and apply confidence intervals and tests to infer its likely value.