Binary Classification
Definition
Binary classification is the task of classifying inputs into two possible categories (classes).
- Classes are often labeled as 0 and 1, or negative and positive.
- Example tasks:
- Spam (1) vs Not Spam (0)
- Disease Present (1) vs No Disease (0)
- Churn (1) vs Retain (0)
Model Output
- Models usually output a probability $p \in [0,1]$ that the instance belongs to the positive class.
- Then a decision threshold (commonly 0.5) is applied:
- $\hat{y} = \begin{cases} 1 & \text{if } p \geq 0.5 \\ 0 & \text{if } p < 0.5 \end{cases}$
Common Algorithms
- Logistic Regression – linear, interpretable, outputs probability.
- Decision Tree – rule-based, easy to interpret.
- Random Forest – ensemble of trees, better accuracy.
- Support Vector Machine (SVM) – finds optimal separating hyperplane.
- k-Nearest Neighbors (kNN) – assigns class by neighbor majority vote.
- Neural Networks – powerful for complex data (e.g., images, text).
- Naïve Bayes – probabilistic, good for text classification.
Evaluation Metrics
Because binary classification often deals with imbalanced data, metrics beyond accuracy are important:
- Confusion Matrix: TP, TN, FP, FN
- Accuracy: $(TP+TN)/(TP+TN+FP+FN)$
- Precision: $TP/(TP+FP)$ (How many predicted positives are correct)
- Recall (Sensitivity): $TP/(TP+FN)$ (How many actual positives are found)
- F1 Score: Harmonic mean of precision and recall
- ROC-AUC: Performance across thresholds
- PR-AUC: Important when positives are rare
Example
Suppose we predict whether an email is spam.
- Dataset: 1000 emails
- Actual spam: 200
- Model predicts spam: 220 (180 correct, 40 wrong)
- Precision = 180/220 = 81.8%
- Recall = 180/200 = 90%
- F1 = 85.7%
This shows the model is good at catching spam while keeping errors relatively low.
Applications
- Healthcare: Predicting disease presence.
- Finance: Fraud detection.
- Marketing: Churn prediction, ad click prediction.
- Security: Intrusion detection.
In short:
Binary classification = predicting between two classes.
Models output probabilities → threshold decides class.
Evaluation uses precision, recall, F1, AUC instead of just accuracy.