Numpy - Basic Linear Algebra Exercise

1. Introduction

Probability and the Naive Bayes classifier are foundational topics in data science, machine learning, and statistics. A solid understanding of probability theory enables you to build powerful predictive models, especially classification models like Naive Bayes – a renowned algorithm known for its simplicity, efficiency, and ease of implementation.

2. Probability Theory

2.1. Definition of Probability

• Probability is a number that measures the likelihood of an event, ranging from 0 to 1.

• 0: The event cannot occur.

• 1: The event is certain to occur.

Example:

Rolling a standard six-sided die, the probability of getting an even number:

$$P(A) = \frac{3}{6} = 0.5$$

Randomly selecting a person who likes AI from a class of 20, with 6 AI enthusiasts:

$$P(A) = \frac{6}{20} = 0.3$$

2.2. Types of Probability

a) Classical Probability

Used when the sample space is finite and all outcomes are equally likely.

$$P(A) = \frac{number\ of\ favorable\ outcomes}{total\ number\ of\ possible\ outcomes}$$

b) Geometric Probability

Applied when the sample space is continuous (measured by length, area, or volume):

$$P(A) = \frac{measure\ of\ domain A}{measure\ of\ the\ entire\ sample\ space}$$

Example:

• Randomly choosing a point on a segment, the probability that the point is closer to 0 than to 1 is 1.53=0.531.5=0.5.

2.3. Basic Rules of Probability

Rule 1:0≤P(A)≤1 Rule 2:The sum of probabilities for all events in the sample space is 1. Rule 3 (Addition):P(A+B)=P(A)+P(B)−P(AB) Rule 4 (Conditional):P(A|B)=P(A∩B)P(B) Rule 5 (Multiplication):P(AB)=P(A)⋅P(B|A) Rule 6 (Independence):If A and B are independent, P(AB)=P(A)⋅P(B) l

2.4. Conditional Probability

The probability of A occurring given that B has occurred:

$$P(A|B) = \frac{P(A \cap B)}{P(B)} \\$$

Example:

In a class of 20 students, 7 like Math, 6 like AI, and 2 like both. The probability of selecting someone who likes Math, given they like AI:

$$P(\text{Math} \mid \text{AI}) = \frac{P(\text{Math} \cap \text{AI})}{P(\text{AI})} = \frac{\dfrac{2}{20}}{\dfrac{6}{20}} = \frac{2}{20} \times \frac{20}{6} = \frac{2}{6}$$

3. Bayes’ Theorem

Bayes’ theorem allows you to update the probability of a hypothesis based on new evidence:

$$P(A \mid H) = \frac{P(H \mid A) \cdot P(A)}{P(H)}$$

Real-world Application:

Suppose a factory produces light bulbs, with 90% being good. The probability a good bulb passes quality control is 0.9, while a defective bulb passes with probability 0.05.

$$\begin{align*} &\text{Probability a bulb passes QC:} \\ &\quad 0.9 \times 0.9 + 0.1 \times 0.05 = 0.815 \\[1.5em] &\text{Probability a defective bulb passes QC:} \\ &\quad \frac{0.1 \times 0.05}{0.815} \approx 0.0061 \end{align*}$$

4. Classification Problem & Naive Bayes Classifier

4.1. The Classification Problem

• Input: A training set of samples with features and class labels.

• Output: Predict the class label for a new sample based on the posterior probability

$$P(c \mid X)$$

4.2. Principle of Naive Bayes

Naive Bayes is based on Bayes’ theorem and assumes features are conditionally independent given the class:

$$\begin{align*} P(c|X) &\propto P(X|c) \cdot P(c) \\[1.5em] \text{With } X = (x_1, x_2, ..., x_N): \\[1.5em] P(X|c) &= P(x_1|c) \cdot P(x_2|c) \cdot \ldots \cdot P(x_N|c) \end{align*}$$

4.3. Steps to Build a Naive Bayes Classifier

a) Training Phase:

• Calculate the prior probability P(c) for each class.

• Compute the conditional probabilities P( x_j | c) for each feature.

b) Prediction Phase:

• For a new sample, compute P(c | X) for each class.

• Assign the class with the highest probability.

5. Practical Example: The “Play Tennis” Problem

5.1. Training Data

5.2. Calculating Probabilities

5.3. Prediction for a New Sample

Suppose the new sample is: Outlook=Sunny, Temperature=Cool, Humidity=High, Wind=Strong

Calculate:

Conclusion: This sample should be classified as “No” since it has the higher probability.

6. Naive Bayes for Continuous Data

When features are real-valued, Naive Bayes assumes the values follow a Gaussian (normal) distribution:

Where μ and σ are the mean and standard deviation of the feature for each class.

Example: IRIS Flower Classification

a) Compute mean and variance for each class

b) Predict for sample X=3.4:

$$\begin{align*} P(\text{Class} = 0 \mid X) &\propto 2.18 \times 10^{-6} \\ P(\text{Class} = 1 \mid X) &\propto 0.697 \end{align*}$$

Conclusion: The sample belongs to class 1, as it has the higher probability.

7. Real-World Applications of Naive Bayes

• Spam email classification

• Weather forecasting

• Medical diagnosis

• Text sentiment analysis

• Text/news/comment classification

• Simple-feature image recognition

8. Advantages and Disadvantages of Naive Bayes

Advantages

• Simple, easy to implement, and scalable for large datasets.

• Effective with high-dimensional data; fast training and prediction.

• Requires little training data.

Disadvantages

• The independence assumption between features is often unrealistic in real data.

• Performs poorly when features are correlated.

• Less effective with rare features or non-Gaussian distributions.

9. Hands-on: Building Naive Bayes in Python

9.1. Basic Steps

1. Data preprocessing: Normalize and encode categorical/continuous features.

2. Calculate prior probabilities: Count samples per class divided by total samples.

3. Calculate conditional probabilities: Count feature occurrences per class, divided by class sample count.

4. Prediction: Multiply conditional probabilities with prior, choose the class with the highest value.

9.2. Basic Pseudocode

# Training phase
for c in classes:
    P[c] = count(c) / total
    for feature in features:
        for value in values:
            P[value|c] = count(value, c) / count(c)

# Prediction phase
for c in classes:
    prob = P[c]
    for feature in features:
        prob *= P[value|c]
    # choose c with the largest prob

10. Conclusion & Advice

Naive Bayes is a powerful, intuitive, and efficient algorithm for many basic classification tasks. However, to achieve optimal performance, you should check the independence assumption between features and consider advanced variants if necessary. Practice with datasets like “Play Tennis” or IRIS to deepen your understanding of how Naive Bayes works