Part of the series Learn TensorFlow Now
In the last post, we saw our network achieve about 60% accuracy. One common way to improve a neural network’s performance is to make it deeper. Before we start adding layers to our network, it’s worth taking a moment to explore one of the key advantages of deep neural networks.
Historically, a lot of effort was invested in crafting hand-engineered features that could be fed to shallow networks (or other learning algorithms). In image detection we might modify the input to highlight horizontal or vertical edges. In voice recognition we might filter out noise or various frequencies not typically found in human speech. Unfortunately, hand-engineering features often required years of expertise and lots of time.
Below is a network created with TensorFlow Playground that demonstrates this point. By feeding modified versions of the input to a shallow network, we are able to train it to recognize a non-linear spiral pattern.
A shallow network is capable of learning complex patterns only when fed modified versions of the input. A key idea behind deep learning is to do away with hand-engineered features whenever possible. Instead, by making the network deeper, we can convince the network to learn the features it really needs to solve the problem. In image recognition, the first few layers of the network learn to recognize simple features (eg. edge detection), while deeper layers respond to more complex features (eg. human faces). Below, we’ve made the network deeper and removed all dependencies on additional features.
Making our network deeper
Let’s try making our network deeper by adding two more layers. We’ll replace layer1_weights and layer1_bias with the following:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| layer1_weights = tf.Variable(tf.random_normal([784, 500])) | |
| layer1_bias = tf.Variable(tf.zeros([500])) | |
| layer1_output = tf.nn.relu(tf.matmul(input, layer1_weights) + layer1_bias) | |
| layer2_weights = tf.Variable(tf.random_normal([500, 500])) | |
| layer2_bias = tf.Variable(tf.zeros([500])) | |
| layer2_output = tf.nn.relu(tf.matmul(layer1_output, layer2_weights) + layer2_bias) | |
| layer3_weights = tf.Variable(tf.random_normal([500, 10])) | |
| layer3_bias = tf.Variable(tf.zeros([10])) | |
| logits = tf.matmul(layer2_output, layer3_weights) + layer3_bias |
Note: When discussing the network’s shapes, I ignore the batch dimension. For example, where a shape is [None, 784] I will refer to it as a vector with 784 elements. I find it helps to imagine a batch size of 1 to avoid having to think about more complex shapes.
The first thing to notice is the change in shape. layer1 now accepts an input of 784 values and produces an intermediate vector layer1_output with 500 elements. We then take these 500 values through layer2 which also produces an intermediate vector layer2_output with 500 elements. Finally, we take these 500 values through layer3 and produce our logit vector with 10 elements.
Why did I choose 500 elements? No reason, it was just an arbitrary value that seemed to work. If you’re following along at home, you could try adding more layers or making them wider (ie. use a size larger than 500).
ReLU
Another important change is the addition of tf.nn.relu() in layer1 and layer2. Note that it is applied to the result of the matrix multiplication of the previous layer’s output with the current layer’s weights.
So what is a ReLU? ReLU stands for “Rectified Linear Unit” and is an activation function. An activation function is applied to the output of each layer of a neural network. It turns out that if we don’t include activation functions, it can be mathematically shown (by people much smarter than me) that our three layer network is equivalent to a single layer network. This is obviously a BadThing™ as it means we lose all the advantages of building a deep neural network.
I’m (very obviously) glossing over the details here, so if you’re new to neural networks and want to learn more see: Why do you need non-linear activation functions?
Other historical activation functions include sigmoid and tanh. These days, ReLU is almost always the right choice of activation function and we’ll be using it exclusively for our networks.
Learning Rate
Finally, one other small change needs to be made: The learning rate needs to be changed from 0.01 to 0.0001. Learning rate is one of the most important, but most finicky hyperparameters to choose when training your network. Too small and the network takes a very long time to train, too large and your network doesn’t converge. In later posts we’ll look at methods that can help with this, but for now I’ve just used the ol’ fashioned “Guess and Check” method until I found a learning rate that worked well.
Alchemy of Hyperparameters
We’ve started to see a few hyperparameters that we must choose when building a neural network:
- Number of layers
- Width of layers
- Learning rate
It’s an uncomfortable reality that we have no good way to choose values for these hyperparameters. What’s worse is that we typically can’t explain why a certain hyperparameter value works well and others do not. The only reassurance I can offer is:
- Other people think this is a problem
- As you build more networks, you’ll develop a rough intuition for choosing hyperparameter values
Putting it all together
Now that we’ve chosen a learning rate and created more intermediate layers, let’s put it all together and see how our network performs.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import tensorflow as tf | |
| from tensorflow.examples.tutorials.mnist import input_data | |
| mnist = input_data.read_data_sets('MNIST_data', one_hot=True) | |
| train_images = mnist.train.images; | |
| train_labels = mnist.train.labels | |
| test_images = mnist.test.images; | |
| test_labels = mnist.test.labels | |
| graph = tf.Graph() | |
| with graph.as_default(): | |
| input = tf.placeholder(tf.float32, shape=(None, 784)) | |
| labels = tf.placeholder(tf.float32, shape=(None, 10)) | |
| #Add our three layers | |
| layer1_weights = tf.Variable(tf.random_normal([784, 500])) | |
| layer1_bias = tf.Variable(tf.zeros([500])) | |
| layer1_output = tf.nn.relu(tf.matmul(input, layer1_weights) + layer1_bias) | |
| layer2_weights = tf.Variable(tf.random_normal([500, 500])) | |
| layer2_bias = tf.Variable(tf.zeros([500])) | |
| layer2_output = tf.nn.relu(tf.matmul(layer1_output, layer2_weights) + layer2_bias) | |
| layer3_weights = tf.Variable(tf.random_normal([500, 10])) | |
| layer3_bias = tf.Variable(tf.zeros([10])) | |
| logits = tf.matmul(layer2_output, layer3_weights) + layer3_bias | |
| cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels)) | |
| #Use a smaller learning rate | |
| learning_rate = 0.0001 | |
| optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) | |
| predictions = tf.nn.softmax(logits) | |
| correct_prediction = tf.equal(tf.argmax(labels, 1), tf.argmax(predictions, 1)) | |
| accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) | |
| with tf.Session(graph=graph) as session: | |
| tf.global_variables_initializer().run() | |
| num_steps = 5000 | |
| batch_size = 100 | |
| for step in range(num_steps): | |
| offset = (step * batch_size) % (train_labels.shape[0] – batch_size) | |
| batch_images = train_images[offset😦offset + batch_size), :] | |
| batch_labels = train_labels[offset😦offset + batch_size), :] | |
| feed_dict = {input: batch_images, labels: batch_labels} | |
| _, c, acc = session.run([optimizer, cost, accuracy], feed_dict=feed_dict) | |
| if step % 100 == 0: | |
| print("Cost: ", c) | |
| print("Accuracy: ", acc * 100.0, "%") | |
| #Test | |
| num_test_batches = int(len(test_images) / 100) | |
| total_accuracy = 0 | |
| total_cost = 0 | |
| for step in range(num_test_batches): | |
| offset = (step * batch_size) % (train_labels.shape[0] – batch_size) | |
| batch_images = test_images[offset😦offset + batch_size), :] | |
| batch_labels = test_labels[offset😦offset + batch_size), :] | |
| feed_dict = {input: batch_images, labels: batch_labels} | |
| _, c, acc = session.run([optimizer, cost, accuracy], feed_dict=feed_dict) | |
| total_cost = total_cost + c | |
| total_accuracy = total_accuracy + acc | |
| print("Test Cost: ", total_cost / num_test_batches) | |
| print("Test accuracy: ", total_accuracy * 100.0 / num_test_batches, "%") |
After running this code you should see output similar to:
Cost: 4596.864 Accuracy: 7.999999821186066 % Cost: 882.4881 Accuracy: 30.000001192092896 % Cost: 609.4177 Accuracy: 51.99999809265137 % Cost: 494.5303 Accuracy: 56.00000023841858 % ... Cost: 57.793114 Accuracy: 89.99999761581421 % Cost: 148.92995 Accuracy: 81.00000023841858 % Cost: 67.42319 Accuracy: 89.99999761581421 % Test Cost: 107.98408660641905 Test accuracy: 85.74999994039536 %
Our network has improved from 60% accuracy to 85% accuracy. This is great progress, clearly things are moving in the right direction! Next week we’ll look at a more complicated neural network structure called a “Convolutional Neural Network” which is one of the basic building blocks of today’s top image classifiers.
For the sake of completeness, I’ve included a TensorBoard visualization of the network we’ve created below:
Shotgun Debugging 