Neural networks: many simple units together

Logistic regression from lesson 5 can only draw a straight boundary. Trees can draw only boxes. But what if the classes curl around each other? You could try to invent the right curve by hand for every problem — or you could build a model out of pieces so simple that each one can only draw a straight line, and let the combination draw anything. That is the bet behind neural networks.

Try this

• Set the hidden-units slider to its smallest value and press Train. Look at the decision regions: how many straight edges can you see?
• Increase the number of hidden units and press Train again. Watch the boundaries bend and compare the accuracy.
• Set the hidden-units slider to its largest value and train. Do the regions start to bend around individual flowers? Ask yourself whether a new flower in such a pocket would really belong to that species.
• Look at the flowers where two species overlap. Can any boundary separate them using only these two measurements?

Now that you've tried it, let's see what was actually going on.

The idea in plain words

A neural network is made of small units called neurons. Each neuron does something we already know: it takes a weighted sum of its inputs, adds an intercept, and passes the result through a simple bending function called an activation. One neuron on its own is nearly logistic regression; it can separate the plane with one straight line.

Neurons are arranged in layers. The inputs (here, two flower measurements) feed a hidden layer of neurons, each of which draws its own line and reports how far, and on which side, a point lies. The output layer combines those reports into the final decision. Because each hidden neuron bends its signal, combining them produces boundaries with corners and curves.

Picture a team of very simple lookouts guarding a field. Each lookout can only say "the intruder is on my left" or "on my right" of a single straight fence. One lookout is not much use. But a supervisor who listens to many lookouts, each with a fence in a different place, can pin down regions of almost any shape.

Why it works

One neuron computes

output = f(w · x + b)

where x is the list of inputs, w the weights, b the intercept, and f the activation. Without f, stacking layers would be pointless: a weighted sum of weighted sums is still just one weighted sum, so the whole network would draw a single straight line. The activation is the crease that stays. With it, each hidden neuron adds one bend, and enough bends can approximate any reasonable boundary. More hidden units mean more possible bends — more flexibility, which, as lesson 7 taught, cuts both ways.

The weights are learned with the tools from lessons 4 and 5. The network makes predictions, the log loss measures how wrong they are, and gradient descent nudges every weight downhill. The only new trick is how to find the gradient for weights deep inside the network: an algorithm called backpropagation applies the chain rule of calculus backwards from the output, layer by layer, telling each weight how much it contributed to the error.

The example in this lesson is the Iris dataset: 150 flowers of three species, 50 of each, made famous by Ronald Fisher's 1936 paper. We take two of the four measurements, so every flower is a point on a plane, and the trained network assigns a predicted species to every spot on that plane.

The idea is old — Frank Rosenblatt built the perceptron, a single trainable neuron, in 1958 — but networks became dominant only when data, computing power and training tricks grew large enough to train many layers.

Common mistake

"Neural" suggests a brain, and that suggestion misleads. An artificial neuron is a weighted sum with a bend, not a model of a biological cell, and a network does not understand flowers; it fits a function to examples. The practical mistake is to assume bigger is always better. A network with many hidden units on a small dataset can memorise exactly as a deep tree or a high-degree polynomial does. Its size has to be chosen with validation data, and its training watched for the same gap between training and test performance we met in lesson 7.

Stack enough layers, feed them enough text, and one prediction task turns into something that writes. In the final lesson we see how a network learns language by guessing the next word.

Want to go deeper? The NN course starts from a single neuron (PL).

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X = X[:, :2]  # sepal length and sepal width, as in the lab
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.5, stratify=y, random_state=1)
for h in (2, 32):
    net = make_pipeline(StandardScaler(), MLPClassifier(hidden_layer_sizes=(h,), activation="tanh", solver="lbfgs", max_iter=5000, random_state=0))
    net.fit(X_tr, y_tr)
    print(f"{h:2d} hidden units: train {net.score(X_tr, y_tr):.1%}, test {net.score(X_te, y_te):.1%}")

 2 hidden units: train 84.0%, test 74.7%
32 hidden units: train 96.0%, test 66.7%