Backpropagation - The How

Introduction

In Part 6, we understood what Backpropagation is and how it works mathematically. We traced the entire process from forward propagation to gradient computation using the chain rule. We learned how errors flow backward through the network and how gradients are calculated for every trainable parameter.

"Part 6 mein humne samjha ki Backpropagation kya hai aur kaise kaam karta hai."

Now it's time to get our hands dirty. In this part, we will implement Backpropagation from scratch and also see how it works in practice using TensorFlow/Keras.

We will cover two main approaches:

1. Manual Implementation using NumPy — Understanding every line of code

2. Implementation using TensorFlow/Keras — How professionals build models

Understanding both approaches is important. The manual implementation explains how neural networks learn, while TensorFlow demonstrates how neural networks are trained in practice.

Before we begin, if you haven't read Part 6 yet, I highly recommend going through it first. The mathematics and gradients we derived there will be used directly in our implementation.

What We Will Cover in This Blog

Now let's begin with a regression example that we started in Part 6.

Example Regression Problem

Suppose we want to predict a student's placement package based on two features.

This is the same dataset we used in Part 6. Here:

• Input features: x₁ = CGPA, x₂ = Resume Score

• Target: y = Package (Lakhs Per Annum)

Network Architecture

The network architecture is simple:

• Input Layer: 2 neurons (CGPA, Resume Score)

• Hidden Layer: 2 neurons

• Output Layer: 1 neuron

Since this is a regression problem, every neuron uses a Linear Activation Function (f(x) = x).

The Complete Training Algorithm

Before writing any code, let's understand the complete training algorithm:

Initialize Weights and Biases

For each Epoch:

For each Training Sample:

    1. Forward Propagation

    2. Calculate Loss

    3. Backpropagation (Compute Gradients)

    4. Update Weights and Biases

Calculate Average Loss for the Epoch

Stop when loss converges (reaches minimum)

One complete iteration over the entire dataset is called an Epoch. Training continues until the loss becomes sufficiently small.

Manual Implementation: Regression

Now let's implement everything step by step using NumPy.

Step 1: Import Libraries and Prepare Data

Step 2: Initialize Parameters

Every layer requires weights and biases. For our network with layer dimensions [2, 2, 1] (input → hidden → output):

• W1: Weights from input to hidden layer (shape: 2×2)

• b1: Bias for hidden layer (shape: 1×2)

• W2: Weights from hidden to output layer (shape: 2×1)

• b2: Bias for output layer (shape: 1×1)

Step 3: Forward Propagation

For every layer, we compute:

Z = WᵀA + b

Since we use linear activation, A = Z (no activation function).

Step 4: Calculate Loss

Since this is a regression problem, we use Mean Squared Error (MSE):

L = (y - ŷ)²

For multiple samples:

MSE = (1/n) × Σ(yᵢ - ŷᵢ)²

Step 5: Backpropagation

Now we implement the gradients we derived in Part 6.

Output Layer Gradients:

∂L/∂W₂ = -2(y - ŷ) × A₁

∂L/∂b₂ = -2(y - ŷ)

Hidden Layer Gradients:

∂L/∂W₁ = -2(y - ŷ) × W₂ × X

∂L/∂b₁ = -2(y - ŷ) × W₂

Step 6: Update Parameters

Using Gradient Descent:

θ_new = θ_old - η × ∂L/∂θ

Where η (eta) is the learning rate.

Step 7: Complete Training Loop

Now we combine everything into a complete training function.

Step 8: Make Predictions

You can run the code of manual implementation in Google Colab using the notebook below:

https://colab.research.google.com/drive/1C1coX-RnNIR6ji3sm6ajkZ2a3LHzQOSm?usp=sharing

---

Implementing the Same Network Using TensorFlow/Keras

You can run the code of keras implementation in Google Colab using the notebook below:

https://colab.research.google.com/drive/11oBqtL56a2bquAxbVEgVWjsx3g9G4nYI?usp=sharing

Backpropagation for Classification

Now let's see how Backpropagation works for classification problems. Classification predicts categories instead of numbers.

Example Dataset

Inputs:

• x₁ = Study Hours

• x₂ = Attendance

Output:

• y ∈ {0, 1} (0 = Fail, 1 = Pass)

Network Architecture

Activation Functions

Hidden Layer: ReLU

ReLU(x) = max(0, x)

Output Layer: Sigmoid

σ(x) = 1 / (1 + e⁻ˣ)

Output:

ŷ = σ(z)

Where 0 ≤ ŷ ≤ 1 represents the probability of the positive class.

Manual Implementation: Classification

Let's implement classification manually using NumPy.

Step 1: Prepare Data

Step 2: Activation Functions

Step 3: Initialize Parameters

Step 4: Forward Propagation

Step 5: Loss Function

For classification, we use Binary Cross Entropy:

L = -[y × log(ŷ) + (1-y) × log(1-ŷ)]

Step 6: Backpropagation

For Binary Cross Entropy combined with Sigmoid, the derivative simplifies beautifully:

∂L/∂z = ŷ - y

This is one of the most important results in deep learning.

Step 7: Complete Training Loop

Step 8: Make Predictions

You can run the code of manual implementation in Google Colab using the notebook below:

https://colab.research.google.com/drive/1RD1uZb3dl-FxgOuprhrKunOk4O1f3x8n?usp=sharing

TensorFlow/Keras Implementation (Binary Classification)

You can run the code of keras implementation in Google Colab using the notebook below:

https://colab.research.google.com/drive/1dlBmPOLaYbaCKMkhPfy755XXNmtQY7RD?usp=sharing

Regression vs Classification

Manual Implementation vs TensorFlow/Keras

The Complete Backpropagation Workflow

The training pipeline is identical for both regression and classification. Only the activation and loss functions differ.

"Yeh hai pura Backpropagation ka workflow."

What's Next?

In this part, we implemented Backpropagation both manually and using Keras. We saw how the same algorithm works for regression and classification problems.

"Ab humne Backpropagation ko implement karke dekha."

But we still haven't answered the most important question: Why does Backpropagation work so well? Why does propagating errors backward and updating weights using gradient descent actually lead to learning?

In Part 8, we will explore The Why of Backpropagation. We will understand:

• The mathematical intuition behind why Backpropagation works

• Why deep networks sometimes fail (vanishing and exploding gradients)

• The theoretical guarantees of gradient descent

• Why Backpropagation is the cornerstone of modern deep learning

Get ready to understand the deeper mathematics behind neural network training. See you in Part 8!