Frequentist

1. Definition

• Frequentist statistics is a framework for statistical inference in which probability is interpreted through the long-run behavior of repeated random experiments or sampling procedures.
• Model parameters, such as a population mean or conversion rate, are treated as fixed but unknown.
• Observed data, estimators, and test statistics are treated as random because they could vary across repeated samples.
• Uncertainty is evaluated using the sampling distributions of estimators and test statistics under specified model assumptions.

---

2. Key Principles

• Parameters are fixed: A population parameter has one true but unknown value.
• Samples are random: Different random samples can produce different estimates.
• Repeated-sampling interpretation: Statistical procedures are evaluated by how they perform across hypothetical repetitions of the sampling process.
• Long-run error control: Procedures can be designed to control quantities such as Type I error rates, confidence interval coverage, and statistical power.

---

3. Common Frequentist Tools

• Point estimate: A single value used to estimate an unknown parameter, such as the sample mean.
• Confidence interval: A range produced by a procedure designed to contain the true parameter at a specified rate across repeated samples, assuming the model conditions hold.
• Hypothesis test: A procedure that evaluates how compatible the observed data are with a null hypothesis.
• P-value: Assuming the null hypothesis and other model assumptions are true, the probability of obtaining a test statistic at least as extreme as the observed one.
• Maximum Likelihood Estimation (MLE): A method that selects parameter values that maximize the likelihood of the observed data.

A p-value is not the probability that the null hypothesis is true, and it does not measure the size or practical importance of an effect.

---

4. Example: A/B Testing

Suppose a company compares the conversion rates of two website versions:

• Version A: Existing design
• Version B: New design

A two-sided frequentist test may use:

$H_0: p_A = p_B$

$H_1: p_A \neq p_B$

If the p-value is below a preselected significance level such as $\alpha = 0.05$, the result provides evidence against the null hypothesis, and the null hypothesis may be rejected.

This does not prove that the versions differ or that Version B is better. The estimated difference, confidence interval, test assumptions, sample size, and practical importance should also be considered.

If the research question specifically asks whether Version B increases conversions, a one-sided alternative may be appropriate when it is defined before examining the results.

---

5. Frequentist vs. Bayesian Statistics

Aspect	Frequentist Statistics	Bayesian Statistics
Probability	Describes long-run frequencies or the repeated-sampling behavior of procedures	Quantifies uncertainty conditional on a statistical model
Parameters	Fixed but unknown	Unknown quantities represented using probability distributions
Data	Random under the assumed sampling process	Observed data are fixed once collected
Prior information	Not represented through a prior probability distribution	Represented through a prior distribution
Primary output	Point estimates, confidence intervals, p-values, and hypothesis tests	Posterior distributions, credible intervals, and posterior probabilities
95% interval interpretation	A procedure that generates 95% confidence intervals will contain the true fixed parameter in 95% of repeated samples under its assumptions	Given the prior, likelihood, model, and observed data, the parameter has a 95% posterior probability of lying within a 95% credible interval
Sequential testing	Repeatedly checking a conventional fixed-sample test can inflate the Type I error rate unless an appropriate sequential method is used	Data can be incorporated sequentially, but the validity and performance of optional stopping depend on the model, prior, stopping rule, and inferential goal

---

6. Advantages

• Provides established methods for controlling long-run error rates.
• Offers widely used tools for experiments, clinical trials, surveys, and A/B testing.
• Does not require the specification of a prior probability distribution.
• Includes mature methods for estimation, testing, model comparison, and experimental design.

---

7. Limitations

• P-values and confidence intervals are frequently misinterpreted.
• A statistically significant result may have little practical importance.
• Results can depend strongly on model assumptions, sampling design, and stopping rules.
• Conventional methods do not directly provide the probability that a hypothesis or parameter value is true.

---

Summary:
Frequentist statistics evaluates statistical procedures through their behavior across repeated samples. Parameters are treated as fixed but unknown, while data and estimators vary across samples. Common frequentist methods include point estimation, confidence intervals, p-values, hypothesis testing, and maximum likelihood estimation.