One-Sided and Two-Sided Tests in a Blind Taste Experiment
A blind taste experiment can be analyzed as a test of a population proportion. Each answer is classified as correct or incorrect, producing binary data similar to coin tosses.
With only 10 trials, however, an exact binomial test is more appropriate than an uncorrected normal z-approximation.
Experimental Setup
Suppose 10 cups are independently filled at random with either Coke or Pepsi. A participant tastes each cup without seeing its identity and identifies 7 correctly.
Define\[ X_i= \begin{cases} 1, & \text{if response }i\text{ is correct},\\ 0, & \text{if response }i\text{ is incorrect}. \end{cases} \]
The total number of correct answers is\[ X=X_1+\cdots+X_{10}. \]
The observed result is\[ X_{\text{obs}}=7. \]
The sample accuracy is\[ \hat p=\frac{7}{10}=0.70. \]
The question is whether 70% accuracy provides convincing evidence of genuine discrimination ability rather than ordinary guessing.
The Null Hypothesis
If each trial has two equally likely alternatives and the participant has no useful information, the probability of a correct guess is\[ p=0.5. \]
The null hypothesis is therefore\[ \boxed{H_0:p=0.5}. \]
Here, \(p\) represents the participant’s probability of correctly classifying a trial under the experimental conditions.
A precise null hypothesis is preferable to the vague phrase “nothing extraordinary is happening.”
The One-Sided Alternative
If the research question is whether the participant performs better than chance, use\[ \boxed{H_A:p>0.5}. \]
This is a one-sided alternative because only results above chance count as evidence for the stated claim.
The hypotheses are\[ H_0:p=0.5 \]
versus\[ H_A:p>0.5. \]
Distribution Under the Null Hypothesis
If the 10 trials are independent and each has probability 0.5 of being classified correctly, then\[ X\mid H_0 \sim \operatorname{Binomial}(10,0.5). \]
The exact probability of \(k\) correct classifications is\[ P(X=k\mid H_0) = \binom{10}{k}(0.5)^k(0.5)^{10-k}. \]
Because both probabilities are 0.5,\[ P(X=k\mid H_0) = \binom{10}{k}(0.5)^{10}. \]
Expected Value and Standard Error
Under \(H_0\),\[ E(X)=np=10(0.5)=5. \]
The standard deviation of the count is\[ SD(X) = \sqrt{np(1-p)}. \]
Therefore,\[ SD(X) = \sqrt{10(0.5)(0.5)} = \sqrt{2.5} \approx 1.581. \]
The observed result is\[ 7-5=2 \]
correct answers above the null expectation.
The Uncorrected z-Approximation
An uncorrected normal statistic is\[ z = \frac{X-np}{\sqrt{np(1-p)}}. \]
Substituting the values gives\[ z = \frac{7-5}{1.581} \approx 1.265. \]
The upper-tail standard normal probability is\[ P(Z\geq1.265) \approx 0.103. \]
This is the source of the approximate 10.2% result.
However, this approximation is poor because:\[ np=5 \]
and\[ n(1-p)=5. \]
The sample is small, and the binomial statistic takes only integer values. An exact binomial calculation should be used.
Exact One-Sided P-Value
For\[ H_A:p>0.5, \]
results at least as favorable to the alternative as seven correct answers are\[ 7,8,9,10. \]
Therefore,\[ \text{p-value} = P(X\geq7\mid p=0.5). \]
Calculate:\[ P(X\geq7) = \sum_{k=7}^{10} \binom{10}{k}(0.5)^{10}. \]
The relevant binomial coefficients are\[ \binom{10}{7}=120, \]\[ \binom{10}{8}=45, \]\[ \binom{10}{9}=10, \]
and\[ \binom{10}{10}=1. \]
Thus,\[ P(X\geq7) = \frac{120+45+10+1}{2^{10}}. \]
Since\[ 2^{10}=1024, \]
we obtain\[ P(X\geq7) = \frac{176}{1024} = 0.171875. \]
Therefore,\[ \boxed{ p_{\text{one-sided}}=0.1719 }. \]
Correct Conclusion
At a 5% significance level,\[ 0.1719>0.05. \]
Therefore, we fail to reject \(H_0\).
The appropriate conclusion is:
Seven correct classifications in 10 trials do not provide strong evidence that the participant’s accuracy exceeds 50%.
This does not prove that the participant lacks discrimination ability. The experiment may simply be too small to distinguish moderate ability from chance variation.
Continuity-Corrected Normal Approximation
If a normal approximation is used, a continuity correction substantially improves it.
The discrete event\[ X\geq7 \]
is approximated by the continuous event\[ Y>6.5. \]
The corrected z-score is\[ z = \frac{6.5-5}{1.581} \approx 0.949. \]
The upper-tail probability is\[ P(Z>0.949) \approx 0.171. \]
This is close to the exact result:\[ 0.1719. \]
The uncorrected value of approximately 10.3% materially understates the p-value.
How Many Correct Answers Would Be Significant?
For an exact one-sided test at level 0.05, examine possible rejection thresholds.
For at least eight correct answers,\[ P(X\geq8) = \frac{ \binom{10}{8} + \binom{10}{9} + \binom{10}{10} }{ 1024 }. \]
Thus,\[ P(X\geq8) = \frac{45+10+1}{1024} = \frac{56}{1024} \approx 0.0547. \]
This is slightly greater than 0.05.
For at least nine correct answers,\[ P(X\geq9) = \frac{10+1}{1024} = \frac{11}{1024} \approx 0.0107. \]
Therefore, an exact nonrandomized test at the 5% level rejects only when\[ \boxed{X\geq9}. \]
With just 10 trials, the test has a coarse rejection rule because the binomial count is discrete.
Two-Sided Alternative
A two-sided alternative is\[ \boxed{ H_A:p\neq0.5 }. \]
This treats both unusually high and unusually low accuracy as evidence against the chance model.
Because the null distribution is symmetric, seven or more correct answers are matched by three or fewer correct answers:\[ |X-5|\geq2. \]
Therefore, the equal-tail exact two-sided p-value is\[ P(X\geq7)+P(X\leq3). \]
By symmetry,\[ P(X\leq3)=P(X\geq7)=0.171875. \]
Thus,\[ \boxed{ p_{\text{two-sided}} = 2(0.171875) = 0.34375 }. \]
The two-sided result is even less significant.
When Very Low Accuracy Might Be Informative
Suppose a participant identifies only one of 10 cups correctly.
Under a two-sided test, this is unusual relative to 50% guessing. A possible explanation is that the participant can detect a difference but consistently reverses the product labels.
However, low accuracy does not automatically prove sensory discrimination. Other possibilities include:
- Misunderstanding the instructions
- Reversing the response buttons
- Recording errors
- A systematic labeling mistake
- Fatigue or response patterns
- Ordinary chance
To establish discrimination with systematic label reversal, the hypothesis and analysis should be planned carefully and ideally confirmed in a new experiment.
Choosing One-Sided or Two-Sided in Advance
The alternative hypothesis must be selected based on the scientific question before examining the outcomes.
Use a one-sided test when:
- Only above-chance accuracy supports the claim.
- Below-chance performance would not be interpreted as evidence in the same direction.
- The direction was chosen before the data were observed.
Use a two-sided test when:
- Any systematic departure from chance matters.
- Both unusually high and unusually low accuracy are scientifically meaningful.
- No direction was justified in advance.
It is invalid to begin with a two-sided test and switch to a one-sided test after seeing that the result falls in a favorable direction.
Why Post Hoc Switching Is Invalid
Suppose the upper-tail probability is\[ 0.03. \]
A prespecified two-sided test might report approximately\[ 0.06. \]
After seeing the result, changing to a one-sided alternative solely to obtain\[ p=0.03 \]
effectively ignores the possibility that the result could have been extreme in the opposite direction.
This inflates the Type I error rate. The nominal 5% threshold no longer describes the true long-run false-positive probability.
Confidence Interval for Accuracy
A point estimate of\[ \hat p=0.70 \]
does not show how uncertain the estimate is.
With only 10 trials, the confidence interval is wide. A Wilson 95% confidence interval is approximately\[ \boxed{ 0.397\text{ to }0.892 }. \]
Thus, the data are compatible with a broad range of accuracies, including values below, near, and substantially above chance.
The interval communicates the limited precision much better than the point estimate of 70% alone.
Experimental Assumptions
The binomial model requires more than simply recording 10 answers.
Equal Chance Under the Null
The participant must genuinely face two equally plausible alternatives on every trial.
Independent Trials
One trial should not determine another. This may be questionable if:
- Tastes carry over between cups
- The participant learns during the experiment
- Fatigue develops
- The sequence contains detectable patterns
- Responses are strategically adjusted
Constant Success Probability
The participant’s accuracy should remain reasonably stable across trials. Temperature, carbonation, palate fatigue, or learning can make \(p\) vary.
Proper Blinding
Neither the participant nor the person interacting with them should unintentionally reveal the cup’s identity.
Randomization
Cup contents and presentation order should be randomized according to a documented procedure.
Comparable Serving Conditions
The products should be served under comparable conditions, including:
- Temperature
- Cup type
- Quantity
- Carbonation
- Timing
- Visibility
- Smell exposure
Exactly Five Cups of Each Product
The binomial model assumes that each cup’s identity is independently randomized with probability 0.5.
If the experiment instead contains exactly five Coke and five Pepsi cups, the cup identities are not independent. Once several identities are known, the remaining composition changes.
The correct null distribution then depends on:
- Whether the participant knows there are exactly five of each
- Whether responses are constrained to five Coke and five Pepsi labels
- How the sequence was randomized
- The participant’s response strategy
A permutation or randomization test may be more appropriate than a simple binomial test.
The experimental design determines the reference distribution.
Repeated Trials from One Participant
Ten answers from one participant do not represent 10 independent people. They represent repeated measurements from one person.
The result can address that participant’s performance under those conditions. It cannot automatically be generalized to:
- All consumers
- All serving conditions
- Future performance
- Other product formulations
For population-level conclusions, many independently sampled participants and an analysis accounting for repeated trials within each participant would be needed.
Improving the Experiment
A stronger design could include:
- More randomized trials
- Palate cleansing between trials
- Balanced presentation orders
- Double blinding
- Identical opaque cups
- Standardized temperature and serving volume
- A preregistered one-sided or two-sided hypothesis
- An exact or design-based analysis
- Multiple participants for population-level inference
- Replication in an independent session
Increasing the number of trials improves statistical power, although dependence and fatigue must be managed.
Key Takeaway
For 7 correct classifications in 10 independent two-choice trials, the null model is\[ X\sim\operatorname{Binomial}(10,0.5). \]
For the prespecified one-sided alternative\[ H_A:p>0.5, \]
the exact p-value is\[ \boxed{ P(X\geq7)=0.1719 }. \]
For the two-sided alternative\[ H_A:p\neq0.5, \]
the equal-tail exact p-value is\[ \boxed{ 0.3438 }. \]
Therefore, seven correct answers do not provide strong evidence of discrimination ability.
The uncorrected z-approximation gives approximately 10.3%, but it is inaccurate for only 10 trials. The exact binomial test—or a continuity-corrected approximation—should be used. Whether the test is one-sided or two-sided must be decided before examining the data, and the validity of the test depends on the experiment’s randomization, blinding, independence, and response design.