Naive Bayes Classification

What you will learn in this lesson:

• Bayes' theorem and what it means to compute "the probability of a class given the features"
• The conditional-independence assumption that makes Naive Bayes naive, and why it's rarely exactly true
• A full worked numeric example classifying a message as spam or not spam
• The three common variants (Gaussian, Multinomial, Bernoulli) and when to use each
• Why Naive Bayes performs surprisingly well despite a clearly false assumption

Bayes' Theorem, in Plain Terms

Naive Bayes is a classifier built directly on top of a single piece of probability theory: Bayes' theorem. It lets you flip a conditional probability around. Instead of asking "given this class, how likely are these features," it lets you answer the question you actually care about: "given these features, how likely is each class."

P(class | features) = P(features | class) · P(class) / P(features)

In words: the probability of a class given the evidence is proportional to how likely that evidence is under that class, multiplied by how common that class is in general. The denominator, P(features), is the same number for every class you're comparing, so in practice you can ignore it and just compare the numerators across classes.

Why "Naive"? The Independence Assumption

Computing P(features | class) directly is hard the moment you have more than one feature, because in general you'd need to know how every feature interacts with every other feature. Naive Bayes sidesteps this entirely by assuming every feature is conditionally independent of every other feature, given the class:

P(x1, x2, ..., xn | class) ≈ P(x1 | class) · P(x2 | class) · ... · P(xn | class)

This assumption is almost never exactly true. In a spam email, the words "free" and "winner" appearing together is more likely than either word's individual probability would suggest if they were truly independent, since spammy emails tend to cluster multiple red-flag words together. The model pretends this clustering doesn't happen.

Despite that, multiplying individual feature likelihoods together is fast, requires very little training data per feature, and in practice often produces rankings between classes that are correct even when the underlying probability estimates themselves are off. That distinction, getting the ranking right even when the exact numbers are wrong, is the main reason this admittedly false assumption still works.

A Worked Example: Spam or Not Spam

Suppose your training data shows that 40% of all emails are spam (so P(spam) = 0.4 and P(not spam) = 0.6). You also know, from counting words in your training set, how often the words "free" and "meeting" appear in each class:

A new message arrives containing both "free" and "meeting." Using the independence assumption, multiply the priors by each word's likelihood under each class:

• Spam score: 0.4 × 0.30 × 0.02 = 0.0024
• Not spam score: 0.6 × 0.05 × 0.20 = 0.0060

The not-spam score is larger, so Naive Bayes classifies this message as not spam, even though it contains the word "free," because "meeting" is a strong enough signal in the other direction and the model combines both pieces of evidence. These raw scores aren't true probabilities (they don't sum to 1) until you divide each by their total, but for picking the winning class, comparing the unnormalized scores directly is enough.

# Reproduce the worked example above directly
p_spam = 0.4
p_not_spam = 0.6

p_free_given_spam = 0.30
p_free_given_not_spam = 0.05
p_meeting_given_spam = 0.02
p_meeting_given_not_spam = 0.20

spam_score = p_spam * p_free_given_spam * p_meeting_given_spam
not_spam_score = p_not_spam * p_free_given_not_spam * p_meeting_given_not_spam

print(f"Spam score:     {spam_score:.4f}")
print(f"Not spam score: {not_spam_score:.4f}")

total = spam_score + not_spam_score
print(f"\nNormalized P(spam | features):     {spam_score / total:.3f}")
print(f"Normalized P(not spam | features): {not_spam_score / total:.3f}")
print(f"\nPredicted class: {'spam' if spam_score > not_spam_score else 'not spam'}")

Three Common Variants

scikit-learn ships three Naive Bayes implementations, each suited to a different kind of feature data:

• GaussianNB: assumes each numeric feature follows a normal distribution within each class. Good for continuous measurements like height, temperature, or sensor readings.
• MultinomialNB: assumes features are counts (like word counts in a document). The standard choice for text classification with a bag-of-words representation.
• BernoulliNB: assumes features are binary (present or absent), which fits a "does this word appear at all" representation rather than a count.

import numpy as np
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

model = GaussianNB()
model.fit(X_train, y_train)

print("Test accuracy:", model.score(X_test, y_test))

# Look at the predicted probabilities for the first 5 test samples,
# not just the hard class label
probs = model.predict_proba(X_test[:5])
print("\nPredicted probabilities [malignant, benign] for first 5 samples:")
print(np.round(probs, 3))

Why It Works Despite Being "Wrong"

Naive Bayes is a useful reminder that a model doesn't need a perfectly accurate assumption to be useful. It only needs the assumption to distort the comparison between classes less than it helps by making the problem tractable. With very little training data, estimating one probability per feature, per class, is far more reliable than trying to estimate every feature interaction directly, which would need exponentially more data as the number of features grows.

This makes Naive Bayes a strong, fast baseline, especially for text classification (spam filtering, sentiment analysis, topic labeling) where the feature count, the vocabulary size, can be in the thousands, and a model that needs to learn pairwise word interactions would need vastly more data than one that treats words as independent given the class.