How studying the brain is helping to build "brains".
Unless you’ve been living under a rock for the last few years, you’ve probably heard of the terms machine learning and neural networks. Maybe you saw the media frenzy surrounding the Go match between GOAT Lee Sedol and the computer program alphaGo (If not, I recommend the excellent documentary about it). Or perhaps you’ve seen some of the ethical debate surrounding artificial intelligence (AI) on the news.
However much you’ve been exposed to these ideas, it doesn’t matter. This post aims to introduce the concepts behind neural networks (a central component of machine learning) in an accessible way by using comparisons with how a real brain functions.
Let’s dive straight in with the concept of a neuron which is where the neural part of Artificial Neural Network (ANN) originates. In the brain and nervous system, neurons are specialised cells that communicate signals with each other via synapses that transmit electrical or chemical signals between neurons. I’ve included a rough diagram of a neuron here (please don’t go and show this to a neuroscientist) to help illustrate some concepts. The diagram may be simple but it shows the branching nature of the head and tail of the neuron. These branches can reach and influence many other neurons via synapses. This branching concept is one that is central to the artificial counterpart of a system of neurons, the ANN.
A gif showing the operation of a ANN is shown below. Look at the features of the ANN. The large circles are called artificial neurons and the lines connecting them are called edges. These edges can be considered as behaving analogously to synapses in that they transfer signals between the (artificial) neurons. Each artificial neuron has an associated number between zero and one called its activation value. In the gif, this activation is shown in orange, a thicker border represents a stronger activation. A neuron with a high activation value can be thought of as sending a stronger signal to connected neurons.
Neurons in a ANN are organised into layers, the first layer is called the input layer and it consists of neurons that receive a signal in the form of external data (i.e. images, text, positions in a chess game etc.). In the gif, the ANN has a cute cat as an input. These can be compared to sensory neurons in the body that respond to external stimuli. The final layer is called the output layer. These neurons output a signal that corresponds to solving the problem at hand( i.e. identifying an image, the vest move to play next etc.). In between the input and output layers are the hidden layers, these hidden layers tend to deal with more abstract representations of the input that help the neural network to calculate the correct output (i.e. certain features in an image, future board states in a chess game etc.). There can be any number of hidden layers in an ANN but it is usually best to use the minimum number possible.
The neurons are linked together between different layers (typically they are not linked within layers) by the edges mentioned earlier. These edges determine how the activation value or signal strength output by neurons in previous layers influences the activation value of the neurons in the next layer. The overall aim of all these connections being the hope that a given set of input neurons firing will be connected with the correct output in the final layer. This is achieved by changing parameters associated with each edge known as weights that specify how strongly the activation value of a neuron affects that of another. In the gif, the edges are thicker if they have a large weight. The activation value of a given neuron is given by a function that takes in the activation values of all of the neurons in the previous layer along with the weights associated with the edges linking those neurons to the current neuron. The function also contains a term known as bias that essentially acts as a threshold ensuring that neurons are only activated if they have received a strong enough signal. The function is of a form such that it takes the signals, weights and bias, and converts them into a number between zero and one which is then the activation value.
For an ANN to work correctly, it is important that the weights and biases are set correctly. The process of determining the correct weights and biases is called training. I will not go into training in detail here because this post is already getting quite long but in brief, it consists of using a large amount of data and specialised techniques that compare what the network is outputting for a given input with what is expected and adjusting the parameters intelligently such that the output moves closer to what is expected.
In conclusion, the way to think of a neural network is like a black-box function that takes an input and gives an output according to parameters which are the weights and biases. What is happening in the black box is what is happening in the hidden layers of the neural network which identify important features of the input to determine the output. The black box will only give the correct outputs for a given input if the values of the parameters are correct which is what is accomplished by training the neural network. This may seem a bit like magic but the training is based on fairly simple mathematical concepts that essentially make sure that the output matches the expected output as well as possible on average. It turns out that neural networks perform outstandingly well and can identify the correct output pretty much all the time.
This post might have seemed a bit of a strange one to include on a statistics and Operational research blog but looking beneath the surface it isn’t. As it turns out many of the techniques in machine learning were originally conceived to solve problems in statistics and mathematical optimization. Not to mention how machine learning looks to be a very promising tool for solving future problems in statistics and OR.
Relevant links
David Silver & Deepmind’s paper on alphaGo
Hello There. I found your weblog using msn. That is a very well written article. I will make sure to bookmark it and come back to read more of your helpful info. Thank you for the post. I’ll definitely return.
Hi. I just stumbled into your page while browsing Yahoo . Ive bookmarked it. Ill definitely be back. I was wondering, have you seen Inception yet? I know this is random. I want to go by the video store and grab it today. Its awesome. See ya.