How Neural Networks Work: A Simple Visual Explanation

Biological Inspiration: From Brain to Computer

Neural networks are inspired by how biological brains work. Our brains contain roughly 86 billion neurons connected by trillions of synapses. Information flows through this network, and learning happens as connections strengthen or weaken based on experience.

Computer scientists created simplified mathematical models of this process, which became artificial neural networks. While inspired by biology, artificial neural networks are far simpler than real brains and work differently in many important ways.

The Artificial Neuron: The Building Block

An artificial neuron is a mathematical function that takes multiple inputs, performs a calculation, and produces an output. Here's how it works:

Step 1: Receive Input

A neuron receives multiple inputs. These might be raw data (pixels in an image) or outputs from previous neurons.

Step 2: Weight the Inputs

Each input is multiplied by a weight. Weights indicate importance: a high weight means that input strongly influences the neuron's output. Low weights mean the input matters little. These weights are what the network learns.

Step 3: Sum and Add Bias

The weighted inputs are summed together. A bias term is added. The bias shifts the output threshold, allowing the neuron to output non-zero values even when all inputs are zero.

Step 4: Apply Activation Function

The result passes through an activation function, which introduces non-linearity. Without activation functions, stacked neurons would just perform linear transformations, unable to learn complex patterns. Common activation functions include ReLU, sigmoid, and tanh.

Step 5: Produce Output

The neuron outputs a single value, which becomes an input to neurons in the next layer or a final prediction.

"A single neuron is just a mathematical function. A network of neurons is powerful because each layer transforms data into increasingly abstract representations."

Network Architecture: Input, Hidden, and Output Layers

Input Layer

The input layer isn't actually neurons: it's just your raw data. If you're classifying images, the input layer has one node per pixel. If you're predicting house prices from features, the input layer has one node per feature (square footage, bedrooms, location, etc.).

Hidden Layers

Hidden layers are where the magic happens. Each hidden layer learns increasingly abstract representations. In image recognition, the first hidden layer might detect edges, the next layer might combine edges into shapes, a deeper layer might recognize eyes and noses, and so on.

More hidden layers allow the network to learn more complex hierarchical representations. However, they also require more data, more computation, and are harder to train. There's no magic formula for the "right" number of layers, it depends on the problem.

Output Layer

The output layer produces final predictions. For binary classification (yes/no), it has one neuron. For multi-class classification (dog, cat, bird), it has one neuron per class. For regression (predicting a number), it typically has one output neuron.

Weights and Biases: Learning the Parameters

Neural networks have two types of learnable parameters: weights and biases. Understanding these is key to understanding how neural networks learn.

Weights: Measuring Importance

A weight indicates how much one input influences a neuron's output. Imagine a doctor diagnosing disease:

• High blood pressure (input) gets a high weight: it's important for heart disease diagnosis
• Hair color (input) gets a low weight, it's irrelevant

Neural networks learn to set weights such that important inputs have high weights and unimportant inputs have low weights.

Biases: Shifting the Threshold

A bias shifts the activation threshold. Without biases, a neuron outputs zero when all inputs are zero, which limits what the network can represent. Biases allow more flexibility.

The Sheer Number of Parameters

A small neural network with 1,000 input neurons, 100 hidden neurons, and 10 output neurons has roughly 110,000 weights and 110 biases: over 110,000 parameters to learn. Large language models have billions of parameters. All of these must be carefully adjusted during training.

Activation Functions: Introducing Non-linearity

Activation functions are essential because they introduce non-linearity. Without them, a neural network would just be a series of linear transformations, capable of learning only linear relationships.

ReLU (Rectified Linear Unit)

The most popular modern activation function. It outputs the input if positive, zero if negative. Simple, computationally efficient, and surprisingly effective.

Sigmoid

Outputs a value between 0 and 1, useful for binary classification. It was popular early on but can suffer from vanishing gradients during training.

Tanh

Outputs a value between -1 and 1. Similar to sigmoid but with better properties for training deep networks.

Softmax

Used in the output layer for multi-class classification. Converts outputs into probabilities that sum to 1.

The Training Process: How Networks Learn

Training is where the magic happens. The network adjusts its weights and biases to minimize prediction errors. Here's the process:

Step 1: Forward Pass

Feed training data through the network. Each neuron performs its calculation, layer by layer. The output layer produces a prediction.

Step 2: Calculate Error

Compare the prediction to the actual answer. How wrong was the network? This is quantified as a loss (error). Lower loss is better.

Step 3: Backward Pass (Backpropagation)

Starting from the output layer and moving backward, calculate how much each parameter (weight and bias) contributed to the error. This uses calculus (specifically, the chain rule) to compute gradients.

Step 4: Update Parameters

Adjust weights and biases in the direction that reduces error. Use gradient descent: if a weight increases error, decrease it; if it decreases error, increase it. The learning rate controls how much to adjust: too high and you overshoot, too low and training is slow.

Step 5: Repeat

Repeat this process thousands or millions of times on batches of training data. With each iteration, the network improves slightly.

Special Network Architectures

Convolutional Neural Networks (CNNs)

Designed for spatial data like images. Instead of fully connected neurons (where each neuron connects to all neurons in the previous layer), CNNs use convolutional layers. Each neuron looks at a small region (e.g., 3x3 pixels) and detects local patterns. This is computationally efficient and allows the network to learn spatial hierarchies: edges, textures, shapes, objects.

CNNs transformed computer vision. They power facial recognition, autonomous vehicles, and medical image analysis.

Recurrent Neural Networks (RNNs)

Designed for sequential data like text or time series. RNNs have connections that loop back, allowing information to persist. They process data sequentially, maintaining hidden state that captures context from previous inputs.

RNNs power language translation, speech recognition, and sentiment analysis. Variants like LSTMs (Long Short-Term Memory) and GRUs (Gated Recurrent Units) address limitations of basic RNNs.

Transformers

A newer architecture that powers modern language models like ChatGPT. Instead of sequential processing, transformers process entire sequences in parallel and use attention mechanisms to determine which parts of the input are most relevant. They've become dominant in natural language processing.