Build a deep neural networks with ReLUs and Softmax.
Deep Neural Networks with TensorFlow

# Deep Learning¶

## Deep Neural Networks¶

Previously we created a pickle with formatted datasets for training, development and testing on the notMNIST dataset.

The goal of this assignment is to progressively train deeper and more accurate models using TensorFlow.

In [15]:
# These are all the modules we'll be using later. Make sure you can import them
# before proceeding further.
from __future__ import print_function
import numpy as np
import tensorflow as tf
from six.moves import cPickle as pickle
from six.moves import range


First reload the data we generated in 1_notmnist.ipynb.

In [16]:
pickle_file = 'notMNIST.pickle'

with open(pickle_file, 'rb') as f:
train_dataset = save['train_dataset']
train_labels = save['train_labels']
valid_dataset = save['valid_dataset']
valid_labels = save['valid_labels']
test_dataset = save['test_dataset']
test_labels = save['test_labels']
del save  # hint to help gc free up memory
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)

Training set (200000, 28, 28) (200000,)
Validation set (10000, 28, 28) (10000,)
Test set (10000, 28, 28) (10000,)


Reformat into a shape that's more adapted to the models we're going to train:

• data as a flat matrix,
• labels as float 1-hot encodings.
In [17]:
image_size = 28
num_labels = 10

def reformat(dataset, labels):
# One shape dimension can be -1.
# In this case, the value is inferred from the length of the array
# and remaining dimensions.
dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)
# Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]
labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
return dataset, labels

train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print('Training set', train_dataset.shape, train_labels.shape)
print('Validation set', valid_dataset.shape, valid_labels.shape)
print('Test set', test_dataset.shape, test_labels.shape)

Training set (200000, 784) (200000, 10)
Validation set (10000, 784) (10000, 10)
Test set (10000, 784) (10000, 10)


Softmax Logistic Regression with Gradient Descent
We're first going to train a multinomial logistic regression using simple gradient descent.

TensorFlow works like this:

• First you describe the computation that you want to see performed: what the inputs, the variables, and the operations look like. These get created as nodes over a computation graph. This description is all contained within the block below:

with graph.as_default():
...
• Then you can run the operations on this graph as many times as you want by calling session.run(), providing it outputs to fetch from the graph that get returned. This runtime operation is all contained in the block below:

with tf.Session(graph=graph) as session:
...

1. Load Data & Build Computation Graph
Let's load all the data into TensorFlow and build the computation graph corresponding to our training:

In [18]:
# With gradient descent training, even this much data is prohibitive.
# Subset the training data for faster turnaround.
train_subset = 10000

# Create graph object: instantiate
graph = tf.Graph()
with graph.as_default():

'''INPUT DATA'''
# Load the training, validation and test data into constants that are
# attached to the graph.
tf_train_dataset = tf.constant(train_dataset[:train_subset, :])
tf_train_labels = tf.constant(train_labels[:train_subset])
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)

'''VARIABLES'''
# These are the parameters that we are going to be training. The weight
# matrix will be initialized using random values following a (truncated)
# normal distribution. The biases get initialized to zero.
weights = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels]))
biases = tf.Variable(tf.zeros([num_labels]))

'''TRAINING COMPUTATION'''
# We multiply the inputs with the weight matrix, and add biases. We compute
# the softmax and cross-entropy (it's one operation in TensorFlow, because
# it's very common, and it can be optimized)
logits = tf.matmul(tf_train_dataset, weights) + biases
# We take the average of this
# cross-entropy across all training examples: that's our loss.
loss = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))

'''OPTIMIZER'''
# We are going to find the minimum of this loss using gradient descent.
# 0.5 is the learning rate

'''PREDICTIONS for the training, validation, and test data.'''
# These are not part of training, but merely here so that we can report
# accuracy figures as we train.
train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases)
test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)


2. Run Computation & Iterate
Let's run this computation and iterate:

In [21]:
num_steps = 801

def accuracy(predictions, labels):
return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
/ predictions.shape[0])

with tf.Session(graph=graph) as session:
# This is a one-time operation which ensures the parameters get initialized as
# we described in the graph: random weights for the matrix, zeros for the
# biases.
tf.initialize_all_variables().run()
print('Initialized')
for step in range(num_steps):
# Run the computations. We tell .run() that we want to run the optimizer,
# and get the loss value and the training predictions returned as numpy
# arrays.
_, l, predictions = session.run([optimizer, loss, train_prediction])
if (step % 100 == 0):
print('Loss at step {}: {}'.format(step, l))
print('Training accuracy: {:.1f}'.format(accuracy(predictions,
train_labels[:train_subset, :])))
# Calling .eval() on valid_prediction is basically like calling run(), but
# just to get that one numpy array. Note that it recomputes all its graph
# dependencies.

# You don't have to do .eval above because we already ran the session for the
# train_prediction
print('Validation accuracy: {:.1f}'.format(accuracy(valid_prediction.eval(),
valid_labels)))
print('Test accuracy: {:.1f}'.format(accuracy(test_prediction.eval(), test_labels)))

Initialized
Loss at step 0: 17.862455368
Training accuracy: 11.7
Validation accuracy: 12.8
Loss at step 100: 2.30228543282
Training accuracy: 72.2
Validation accuracy: 71.3
Loss at step 200: 1.83695435524
Training accuracy: 75.2
Validation accuracy: 73.2
Loss at step 300: 1.58685815334
Training accuracy: 76.4
Validation accuracy: 73.7
Loss at step 400: 1.42111968994
Training accuracy: 77.5
Validation accuracy: 73.9
Loss at step 500: 1.29973900318
Training accuracy: 78.1
Validation accuracy: 74.0
Loss at step 600: 1.20515108109
Training accuracy: 78.6
Validation accuracy: 74.2
Loss at step 700: 1.12850558758
Training accuracy: 79.1
Validation accuracy: 74.3
Loss at step 800: 1.06485939026
Training accuracy: 79.5
Validation accuracy: 74.4
Test accuracy: 82.3


Let's now switch to stochastic gradient descent training instead, which is much faster.

The graph will be similar, except that instead of holding all the training data into a constant node, we create a Placeholder node which will be fed actual data at every call of session.run().

1. Load Data & Build Computation Graph

Placeholders

• tf_train_dataset isn't a specific value.
• It's a placeholder, a value that we'll input when we ask TensorFlow to run a computation.
• We represent this as a 2-D tensor of floating-point numbers, with a shape [batch_size, image_size * image_size]
• If there is None, it means that a dimension can be of any length.
In [22]:
batch_size = 128

graph = tf.Graph()
with graph.as_default():

'''INPUT DATA'''
# For the training data, we use a placeholder that will be fed
# at run time with a training minibatch.
tf_train_dataset = tf.placeholder(tf.float32,
shape=(batch_size, image_size * image_size))
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)

'''VARIABLES'''
# These are the parameters that we are going to be training. The weight
# matrix will be initialized using random values following a (truncated)
# normal distribution. The biases get initialized to zero.
weights = tf.Variable(tf.truncated_normal([image_size * image_size, num_labels]))
biases = tf.Variable(tf.zeros([num_labels]))

'''TRAINING COMPUTATION'''
# We multiply the inputs with the weight matrix, and add biases. We compute
# the softmax and cross-entropy (it's one operation in TensorFlow, because
# it's very common, and it can be optimized)
logits = tf.matmul(tf_train_dataset, weights) + biases
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))

'''OPTIMIZER'''
# We are going to find the minimum of this loss using gradient descent.
# 0.5 is the learning rate

'''PREDICTIONS for the training, validation, and test data'''
# These are not part of training, but merely here so that we can report
# accuracy figures as we train.
train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases)
test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)


2. Run Computation & Iterate

In [0]:
num_steps = 3001

with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print("Initialized")
for step in range(num_steps):
# Pick an offset within the training data, which has been randomized.
# Note: we could use better randomization across epochs.
offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
# Generate a minibatch.
batch_data = train_dataset[offset:(offset + batch_size), :]
batch_labels = train_labels[offset:(offset + batch_size), :]
# Prepare a dictionary telling the session where to feed the minibatch.
# The key of the dictionary is the placeholder node of the graph to be fed,
# and the value is the numpy array to feed to it.
feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
_, l, predictions = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
if (step % 500 == 0):
print("Minibatch loss at step {}: {}".format(step, l))
print("Minibatch accuracy: {:.1f}".format(accuracy(predictions, batch_labels)))
print("Validation accuracy: {:.1f}".format(accuracy(valid_prediction.eval(), valid_labels)))
print("Test accuracy: {:.1f}".format(accuracy(test_prediction.eval(), test_labels)))

Initialized
Minibatch loss at step 0 : 16.8091
Minibatch accuracy: 12.5%
Validation accuracy: 14.0%
Minibatch loss at step 500 : 1.75256
Minibatch accuracy: 77.3%
Validation accuracy: 75.0%
Minibatch loss at step 1000 : 1.32283
Minibatch accuracy: 77.3%
Validation accuracy: 76.6%
Minibatch loss at step 1500 : 0.944533
Minibatch accuracy: 83.6%
Validation accuracy: 76.5%
Minibatch loss at step 2000 : 1.03795
Minibatch accuracy: 78.9%
Validation accuracy: 77.8%
Minibatch loss at step 2500 : 1.10219
Minibatch accuracy: 80.5%
Validation accuracy: 78.0%
Minibatch loss at step 3000 : 0.758874
Minibatch accuracy: 82.8%
Validation accuracy: 78.8%
Test accuracy: 86.1%


Offset Explanation

• The offset gives an arithmetic sequence within each epoch and different offsets can be obtained among different epochs.
• Epoch:
• Measure of the number of times all of the training vectors are used once to update the weights.
• For batch training, all of the training samples pass through the learning algorithm simultaneously in one epoch before weights are updated.
• For sequential training, all of the weights are updated after each training vector is sequentially passed through the training algorithm.
• The expression for the offset generates a cyclic group of numbers.
• These offsets make each mini-batch different from each other not only within each epoch but also among epochs.
• The reason why we randomly shift the batch_data is that if you sample a dataset according to a distribution P enough times, then you can estimate the expectation value for the dataset.
• In other words, you can estimate the loss function over the training dataset by randomly choosing each mini-batch dataset.
• Example: batch_size = 3 and size of train_labels = 100:
• steps = 1,2,...,32
• offset = 3,6,...,96
• steps = 33,34,...,64
• offset = 2,5,..,95
• for steps = 65,66,...,96
• offset = 1,4,..,94, and so on.

## Problem¶

Turn the logistic regression example with SGD into a 1-hidden layer neural network with rectified linear units nn.relu() and 1024 hidden nodes. This model should improve your validation / test accuracy.

In [24]:
num_nodes= 1024
batch_size = 128

graph = tf.Graph()
with graph.as_default():

# Input data. For the training data, we use a placeholder that will be fed
# at run time with a training minibatch.
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, image_size * image_size))
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)

# Variables.
weights_1 = tf.Variable(tf.truncated_normal([image_size * image_size, num_nodes]))
biases_1 = tf.Variable(tf.zeros([num_nodes]))
weights_2 = tf.Variable(tf.truncated_normal([num_nodes, num_labels]))
biases_2 = tf.Variable(tf.zeros([num_labels]))

# Training computation.
logits_1 = tf.matmul(tf_train_dataset, weights_1) + biases_1
relu_layer= tf.nn.relu(logits_1)
logits_2 = tf.matmul(relu_layer, weights_2) + biases_2
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits_2, tf_train_labels))

# Optimizer.

# Predictions for the training
train_prediction = tf.nn.softmax(logits_2)

# Predictions for validation
logits_1 = tf.matmul(tf_valid_dataset, weights_1) + biases_1
relu_layer= tf.nn.relu(logits_1)
logits_2 = tf.matmul(relu_layer, weights_2) + biases_2

valid_prediction = tf.nn.softmax(logits_2)

# Predictions for test
logits_1 = tf.matmul(tf_test_dataset, weights_1) + biases_1
relu_layer= tf.nn.relu(logits_1)
logits_2 = tf.matmul(relu_layer, weights_2) + biases_2

test_prediction =  tf.nn.softmax(logits_2)

In [25]:
num_steps = 3001

with tf.Session(graph=graph) as session:
tf.initialize_all_variables().run()
print("Initialized")
for step in range(num_steps):
# Pick an offset within the training data, which has been randomized.
# Note: we could use better randomization across epochs.
offset = (step * batch_size) % (train_labels.shape[0] - batch_size)
# Generate a minibatch.
batch_data = train_dataset[offset:(offset + batch_size), :]
batch_labels = train_labels[offset:(offset + batch_size), :]
# Prepare a dictionary telling the session where to feed the minibatch.
# The key of the dictionary is the placeholder node of the graph to be fed,
# and the value is the numpy array to feed to it.
feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
_, l, predictions = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
if (step % 500 == 0):
print("Minibatch loss at step {}: {}".format(step, l))
print("Minibatch accuracy: {:.1f}".format(accuracy(predictions, batch_labels)))
print("Validation accuracy: {:.1f}".format(accuracy(valid_prediction.eval(), valid_labels)))
print("Test accuracy: {:.1f}".format(accuracy(test_prediction.eval(), test_labels)))

Initialized
Minibatch loss at step 0: 334.423492432
Minibatch accuracy: 7.8
Validation accuracy: 33.4
Minibatch loss at step 500: 19.1735744476
Minibatch accuracy: 82.0
Validation accuracy: 80.4
Minibatch loss at step 1000: 9.4803943634
Minibatch accuracy: 76.6
Validation accuracy: 80.9
Minibatch loss at step 1500: 6.51924324036
Minibatch accuracy: 87.5
Validation accuracy: 81.6
Minibatch loss at step 2000: 3.05096673965
Minibatch accuracy: 85.2
Validation accuracy: 81.4
Minibatch loss at step 2500: 6.35765266418
Minibatch accuracy: 80.5
Validation accuracy: 80.6
Minibatch loss at step 3000: 2.38029289246
Minibatch accuracy: 83.6
Validation accuracy: 81.8
Test accuracy: 89.0

Tags: