A neural network is a set of computing units arranged in layers, where each connection carries a weight that the network adjusts as it learns from examples. Frameworks like PyTorch hide that learning behind automatic differentiation. In this tutorial we build it by hand with NumPy, so we can watch the forward pass, the loss, and backpropagation work on data.
Key takeaways
A two layer network is four pieces: weights, a forward pass, a loss, and a backward pass that updates the weights.
ReLU in the hidden layer bends the decision boundary, which is what makes XOR solvable at all.
He initialisation scales the starting weights, so training starts stable instead of stuck.
For a sigmoid output with binary cross entropy, the output gradient is simply prediction minus target.
Our hand written network scored 4 out of 4 on XOR and 97.0 percent on held out moons data, in under 4 seconds.
What are we building?
We build one small class, NeuralNetwork, with an input layer, one hidden layer of ReLU units, and a single sigmoid output. We test it twice. First on XOR, the classic problem that no straight line can solve. Then on the two moons data set, 1,000 curved points in two classes that no straight line can separate either. scikit-learn only generates the moons data and splits it. The network itself uses NumPy only. NumPy describes itself as the fundamental package for scientific computing with Python (NumPy docs). The make_moons generator builds two interleaving half circles, a shape that forces the network to learn a curve (scikit-learn docs).
We need Python 3.12 and two packages. We ran this tutorial on Python 3.12.3 with NumPy 2.5.4 and scikit-learn 1.9.1:
python -m venv .venv
source .venv/bin/activate
pip install numpy scikit-learn
mkdir nn-scratch && cd nn-scratch
How does the network store what it learns?
Step 1 is the constructor and the activation helpers. Everything the network knows lives in four arrays: W1 and b1 connect the inputs to the hidden layer, W2 and b2 connect the hidden layer to the output. We start the weights with He initialisation, which scales random values by the square root of 2 divided by the number of inputs. That scale is the standard choice for ReLU layers, and it is what Keras and PyTorch use by default. Biases start at zero. ReLU returns the input when it is positive and zero otherwise, and its derivative is 1 for positive inputs and 0 for the rest. The sigmoid squashes any number into the range 0 to 1, so the output reads as a probability.
import numpy as np
rng = np.random.default_rng(42)
def relu(z):
return np.maximum(0, z)
def relu_derivative(z):
return (z > 0).astype(float)
def sigmoid(z):
return 1 / (1 + np.exp(-z))
class NeuralNetwork:
"""Two-layer network: input -> ReLU hidden -> sigmoid output."""
def __init__(self, n_inputs, n_hidden, lr=0.1):
# He initialisation: scale weights by sqrt(2 / fan_in), the
# standard choice for ReLU layers (Keras and PyTorch default).
self.W1 = rng.standard_normal((n_inputs, n_hidden)) * np.sqrt(2 / n_inputs)
self.b1 = np.zeros((1, n_hidden))
self.W2 = rng.standard_normal((n_hidden, 1)) * np.sqrt(2 / n_hidden)
self.b2 = np.zeros((1, 1))
self.lr = lr
We use np.random.default_rng(42) instead of the older np.random.seed call, so every run in this tutorial gives the same numbers. That matters, because the results below are the exact numbers our run produced.
How does the forward pass work?
Step 2 is the forward pass and the loss. Data flows left to right. We multiply the inputs by W1, add b1, and apply ReLU to get the hidden activations. Then we multiply those by W2, add b2, and apply the sigmoid to get a prediction between 0 and 1. The loss is binary cross entropy, which punishes confident wrong answers far more than unsure ones. We clip the prediction away from exactly 0 and 1 first, because log of zero is undefined.
def forward(self, X):
self.z1 = X @ self.W1 + self.b1
self.a1 = relu(self.z1)
self.z2 = self.a1 @ self.W2 + self.b2
self.a2 = sigmoid(self.z2)
return self.a2
@staticmethod
def loss(y_true, y_pred):
eps = 1e-15
y_pred = np.clip(y_pred, eps, 1 - eps)
return float(-np.mean(y_true * np.log(y_pred)
+ (1 - y_true) * np.log(1 - y_pred)))
How does backpropagation update the weights?
Step 3 is the backward pass, and this is where the learning happens. We work right to left with the chain rule. The neat part is the output layer: for a sigmoid output paired with binary cross entropy, the gradient simplifies to prediction minus target, divided by the number of samples. From there we push the error back through W2, multiply by the ReLU derivative to pass only the active hidden units, and get the gradients for W1 and b1. Then we nudge every weight and bias a small step against its gradient, with the step size set by the learning rate.
def backward(self, X, y):
m = X.shape[0]
# Output layer: for sigmoid + binary cross-entropy, the
# gradient simplifies to (prediction - target) / m.
dz2 = (self.a2 - y) / m
dW2 = self.a1.T @ dz2
db2 = np.sum(dz2, axis=0, keepdims=True)
da1 = dz2 @ self.W2.T
dz1 = da1 * relu_derivative(self.z1)
dW1 = X.T @ dz1
db1 = np.sum(dz1, axis=0, keepdims=True)
self.W1 -= self.lr * dW1
self.b1 -= self.lr * db1
self.W2 -= self.lr * dW2
self.b2 -= self.lr * db2
def train(self, X, y, epochs, log_every=0):
for epoch in range(1, epochs + 1):
preds = self.forward(X)
self.backward(X, y)
if log_every and epoch % log_every == 0:
print(f"epoch {epoch:>5} loss {self.loss(y, preds):.4f}")
return self.loss(y, self.forward(X))
def predict(self, X):
return (self.forward(X) > 0.5).astype(int)
Two details are worth noting. First, the whole data set goes through in one batch each epoch, which keeps the code short and the run fast. Second, nothing here needs a framework. Every line above is plain NumPy, and that is the point: when a bigger model misbehaves later, we will know what each of these steps is supposed to do.
How do we train it, and does it learn XOR?
Step 4 is the first test. XOR outputs 1 only when its two inputs differ, so the four points form a checkerboard that no single straight line can split. A linear model scores 50 percent on it. Our network has 2 inputs, 4 hidden units, and a learning rate of 0.5, trained for 5,000 epochs:
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]], dtype=float)
y_xor = np.array([[0], [1], [1], [0]], dtype=float)
xor_net = NeuralNetwork(n_inputs=2, n_hidden=4, lr=0.5)
final_loss = xor_net.train(X_xor, y_xor, epochs=5000, log_every=1000)
print(f"final loss {final_loss:.4f}")
print("predictions:", xor_net.predict(X_xor).ravel().tolist())
print("expected: ", y_xor.ravel().astype(int).tolist())
The real output from our run:
epoch 1000 loss 0.0048
epoch 2000 loss 0.0020
epoch 3000 loss 0.0013
epoch 4000 loss 0.0009
epoch 5000 loss 0.0007
final loss 0.0007
predictions: [0, 1, 1, 0]
expected: [0, 1, 1, 0]
All four predictions are correct, and the loss falls to 0.0007. The hidden layer is doing the real work here: with ReLU units it can carve the input space into regions, while a single layer could only draw one straight line.
How well does it do on real data?
Step 5 is the harder test, the two moons data set. We generate 1,000 points with a noise level of 0.2, hold out 200 for testing, and train a 2 input, 8 hidden unit network for 2,000 epochs at the same learning rate of 0.5:
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
X, y = make_moons(n_samples=1000, noise=0.2, random_state=42)
y = y.reshape(-1, 1).astype(float)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42)
moon_net = NeuralNetwork(n_inputs=2, n_hidden=8, lr=0.5)
final_loss = moon_net.train(X_train, y_train, epochs=2000, log_every=500)
print(f"final loss {final_loss:.4f}")
print(f"train accuracy {accuracy(y_train, moon_net.predict(X_train)):.4f}")
print(f"test accuracy {accuracy(y_test, moon_net.predict(X_test)):.4f}")
The real output from our run:
train 800 samples, test 200 samples
epoch 500 loss 0.1104
epoch 1000 loss 0.0866
epoch 1500 loss 0.0796
epoch 2000 loss 0.0747
final loss 0.0747
train accuracy 0.9700
test accuracy 0.9700
first 10 test predictions: [1, 0, 1, 0, 1, 1, 0, 0, 1, 0]
first 10 test labels: [1, 0, 1, 0, 1, 1, 0, 0, 1, 0]
Test accuracy is 97.0 percent, the same as train accuracy, so the network is not memorising the training points. The first 10 test predictions all match their labels. The whole file, both problems included, runs in 3.9 seconds on a CPU, which shows how far plain NumPy goes on a small problem.
What breaks when the learning rate is wrong?
Step 6 is the honest check, because a tutorial that only shows the happy path hides the most useful lesson. We reran the moons training at two other learning rates. At 0.05 the network learns too slowly: after 2,000 epochs it reaches only 88.5 percent on train and 89.5 percent on test, with a loss of 0.2478. It is still improving, it just needs far more epochs. At 5.0, ten times our rate, it still reached 96.8 percent on train and 96.5 percent on test. That surprised us: on this small full batch problem a high rate is not always fatal, but it is a gamble, and on larger problems it usually diverges. The rule we take from this is simple: watch the loss curve. If it barely moves, raise the rate. If it jumps around or grows, lower it.
What should we try next?
First, swap the moons data for the handwritten digits data set, which scikit-learn ships built in: 64 inputs, 10 outputs, and a softmax layer instead of one sigmoid. Second, add a second hidden layer and see whether the moons accuracy climbs past 97 percent. Third, replace full batch training with mini batches of 32 samples and shuffle them each epoch, which is how real training loops work and usually learns faster. Each of these is a small change to the one file we already have.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)