Interfacing Tensorflow to Noobs

Hi there!! This post is supplementary to the presentation hosted here and all the codes can be found here.

TensorFlow (TF) is an open-source numerical computation python library maintained by Google Inc. It has enabled long-standing powerful deep learning techniques accessible for both production and research. Owing to its strong community backing and flexible design it has become the mainstay in the AI fraternity. In this post, I’ll walk you through the nitty-gritty of getting started with TF. The audience is expected to be either completely new or beginners to it and who wants to use it either for making any product or do research. So, let’s get started.

Before getting into the main content, let me first explain how we are going to go about getting acquainted with the nuts and bolts of Tensorflow. The entire talk can be divided into four subparts as shown below.

Diagram1

 

At the end of each subpart, you’ll be asked to answer a quiz question and/or summarize your understanding in few lines.

The solutions to these quiz questions are there at the end of this blog post. But don’t scroll down to see the answer without giving your gray cells a stretch. This is for your own benefit, and so unless you have pretty strong reasons to not seek the best, go on and scroll straight away.

Summarizing in a few sentences will help you out to recall the main takeaways and help you feed them into your long-term memory. This critical way of recollecting anything is a scientifically proven way of how to learn anything smartly. You can read more about it here and here. I will also request every one of you to post your summaries here as a comment and read other student’s comments. This will help you to have more outcome from today’s talk.

We’ll first contextualize everything that we’ll cover today that will lay the foundation for further chapters. The second and third subpart will explain the fundamental of TF with a demonstration using Deep Learning (DL) Architectures. The second chapter will make use of higher-level APIs which are easier to use but don’t give a lot of control on your Machine Learning (ML) architecture. Hence, it is not suitable for research. The third chapter will again solve the same problem as in the second chapter but this time will make use finer level controls in TF. Albeit its steep learning curve, this is an extremely useful part to know as an ML enthusiast. The final chapter will interface you with another high-level APIs named Keras which has been very recently merged into TF.

Chapter 1:: Putting things into perspective

Demo problem to be tackled:
For the purpose of talk today, we’ll solve the problem of hand-written image recognition task using DL Architectures whose dataset is hosted as MNIST (Modified National Institute of Standards and Technology”) on Yann LeCunn’s website.

 

 

MNIST has served as a classic dataset for benchmarking different algorithms mainly for computer vision tasks over last few years. The task is to determine digits from handwritten digits.

Abstracting ML model development:
The process of finding an acceptable ML model for solving any task entails an iterative process of three tasks – Making INFERENCE, determining LOSS, and UPDATING the parameters to make better inferences. It is to be noted that by the term parameters, we mean weights and biases of any DL architecture.

Steps

This holds true invariably for all ML tasks. Likewise, we’ll see in next two chapters (chapter 2 and chapter 3) how our entire focus would be in designing ways in TF  to substantiate this abstraction of three steps.

Quiz:
This brings us to our first quiz question. Give your best shot to answer this question. Match the rows in columns 1 with the rows in column 2.

Quiz 1

Summary:
Answer this to finish up chapter 1.

Can the combination of the three abstract steps described above called as an OPTIMIZATION?

 

Chapter 2:: TF using high-level APIs

Its code is here.

TF high-level API is tf.contrib.learn :: helps to

  • manage datasets,
  • estimators,
  • inference, and
  • training.
classifier = tf.contrib.learn.DNNClassifier(feature_columns=feature_columns,
                                              hidden_units=[HIDDEN_LAYER_1, HIDDEN_LAYER_2],
                                              n_classes=NUM_CLASSES,
                                              model_dir="./model_data")

classifier.fit(input_fn=get_train_inputs, steps=2000)

Well, these two lines of code encapsulate all those three steps that we talked about. Albeit, its relative ease of use, it doesn’t give enough flexibility to researchers.

Steps

We’ll see in the next chapter how TF also gives enough power to researchers by giving ways to tune every single detail of the ML architecture.

 

Summary:
Answer this to finish up chapter 2.

How would you customize the inference step as highlighted in the image on the above demonstration to incorporate a new and never used before activation function, say h = y(x, weights, biases)?

 

Infer

Chapter 3:: Finer controls in TF

Codes are here and here .

For understanding the finer controls in TF, we need to understand the mechanism that runs under the hood. Well, what happened in the last demo, TF constructed a highly computationally efficient computation graph on top of which it performs all its calculations. So, essentially TF mechanism follows two steps:

  1. Building a computational graph
  2. Running the computational graph

Let’s take the two different but intertwined processes of computational graph generation and computations separately, quoting the exact lines from the demonstration MNIST code .

Computational Graph Generation:

Like any other computational graph, this graph is also composed of nodes. These nodes make use the following functional units to accomplish its different goals.

  • Tensors::
    It is the most fundamental piece shaped as an array inside TF. Every node in the TF graph does input-output through Tensors. A few examples (taken from https://www.tensorflow.org/get_started/get_started) would be:-- 3 # a rank 0 tensor; this is a scalar with shape []
    -- [1. ,2., 3.] # a rank 1 tensor; this is a vector with shape [3]
    [[1., 2., 3.],
    -- [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]
    -- [[[1., 2., 3.]],
    [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]

 

  • Constants:
    It is a type of node whose value doesn’t change over time. It can be defined as

 

constant_node = tf.constant(3, tf.int64)

where, the first argument is the value that we want the constant node to hold, and the second argument is its datatype. tf.float32 is the default data type of the constant nodes.

  • Placeholders:
    Placeholders are means to accept inputs into the computational graph from the user. They can be defined as

 

placeholder_node = tf.placeholder(tf.float32, shape=[None, 10])

It is to be noted that the second argument in this example shape is optional. But because this is important, I have put it here. The example says the shape of the input should be a 2D array that can contain any number of rows, but each row should contain exactly ten columns. Note that None here means any number of rows is acceptable as input. Can you think of any reason why this might be useful?

These are the lines from our MNIST demonstration code.

# Allocating nodes in the computational graph to accept inputs

images = tf.placeholder(tf.float32, shape=(None, IMAGE_PIXELS))
true_labels = tf.placeholder(tf.int32, shape=(None, NUM_CLASSES))
  • Variables:
    These are the nodes that contain the parametric values of any model. Remember that the parameters of a DL model are its weights and biases. Hence, we define our weights and biases (in fact any new unconventional parameter that you want to test) nodes like the following (from https://www.tensorflow.org/get_started/get_started)

    weight = tf.Variable([0.3], tf.float32)
    bias = tf.Variable([-0.3], tf.float32)

    Note that every variable should be given an initial value as given as first argument in the example. It is also worth noting here that all variables are initialized in the computational graph separately using tf.global_variables_initializer() function which we will be covering in a few moments.
    These are the lines from our MNIST demonstration code.

    # Defining the first hidden layer
    weights_layer_1 = tf.Variable(tf.truncated_normal([IMAGE_PIXELS, HIDDEN_LAYER_1],
                          stddev=1.0 / math.sqrt(float(IMAGE_PIXELS))))
    
    biases_layer_1 = tf.Variable(tf.zeros([HIDDEN_LAYER_1]))
    
    hidden_output_1 = tf.nn.relu(tf.matmul(images, weights_layer_1) + biases_layer_1)
    
    # Defining the second hidden layer
    weights_layer_2 = tf.Variable(tf.truncated_normal([HIDDEN_LAYER_1, HIDDEN_LAYER_2],
                          stddev=1.0 / math.sqrt(float(HIDDEN_LAYER_1))))
    
    biases_layer_2 = tf.Variable(tf.zeros([HIDDEN_LAYER_2]))
    
    hidden_output_2 = tf.nn.relu(tf.matmul(hidden_output_1, weights_layer_2) + biases_layer_2)
    
    # Defining the outputs
    weights_output = tf.Variable(tf.truncated_normal([HIDDEN_LAYER_2, NUM_CLASSES],
                          stddev=1.0 / math.sqrt(float(HIDDEN_LAYER_2))))
    
    biases_output = tf.Variable(tf.zeros([NUM_CLASSES]))
    
    prediction = tf.matmul(hidden_output_2, weights_output) + biases_output

Remember the three abstract steps that we claimed comprised the development of any ML model. Here is it for your reference again.

Steps

Before moving onto how to run computations on the computational graph, let’s break down our demonstration MNIST code to see how we achieved designing these three steps.

Shown below are the portions of our demonstration MNIST code  with their appropriate mapping to the steps they fulfill as highlighted in the images.

  1. # Defining the first hidden layer
    weights_layer_1 = tf.Variable(tf.truncated_normal([IMAGE_PIXELS, HIDDEN_LAYER_1],
                          stddev=1.0 / math.sqrt(float(IMAGE_PIXELS))))
    
    biases_layer_1 = tf.Variable(tf.zeros([HIDDEN_LAYER_1]))
    
    hidden_output_1 = tf.nn.relu(tf.matmul(images, weights_layer_1) + biases_layer_1)

     

    # Defining the second hidden layer
    weights_layer_2 = tf.Variable(tf.truncated_normal([HIDDEN_LAYER_1, HIDDEN_LAYER_2],
                          stddev=1.0 / math.sqrt(float(HIDDEN_LAYER_1))))
    
    biases_layer_2 = tf.Variable(tf.zeros([HIDDEN_LAYER_2]))
    
    hidden_output_2 = tf.nn.relu(tf.matmul(hidden_output_1, weights_layer_2) + biases_layer_2)

     

    # Defining the outputs
    weights_output = tf.Variable(tf.truncated_normal([HIDDEN_LAYER_2, NUM_CLASSES],
                          stddev=1.0 / math.sqrt(float(HIDDEN_LAYER_2))))
    
    biases_output = tf.Variable(tf.zeros([NUM_CLASSES]))
    
    prediction = tf.matmul(hidden_output_2, weights_output) + biases_output

    Infer

  2. # Evaluating the loss function
    loss = tf.reduce_mean(
        tf.nn.softmax_cross_entropy_with_logits(labels=true_labels, logits=prediction))

    Loss

  3. # Updating the parameters
    optimizer = tf.train.GradientDescentOptimizer(0.01)
    training = optimizer.minimize(loss)
    Optimize

 

Graph Computation:
Did anyone see the output of print command in the Fine Controls in TF – 1 file. You’ll see that the output on the nodes doesn’t print their values. For example,
print(placeholder_node)

this gives as output

Tensor("Placeholder_1:0", shape=(?, 10), dtype=float32)
Hence, these nodes are of no use unless we create a session connection to the graph and then run them through that session.
Running computations on a computational graph involves creating a session. A TF session captures the control and state of the computational graph and gives you a handle to connect to the backend computational graph. It can be created either as
sess = tf.Session()
or
sess = tf.InteractiveSession()
Now, we can compute and evaluate the values of all our nodes through this sess handle. However, before computing anything, we have to initialize the variables. We do that by these two lines of code.
init = tf.global_variables_initializer()
sess.run(init)
We can now tell the sess to evaluate and give the values of various nodes as
print(sess.run(constant_node))
which gives us the output as 3.
Let’s make something more useful by making a prediction linear in nature with the inputs.
linear_prediction = weight * placeholder_node + bias
Let’s now evaluate the prediction given some inputs. Inputs can be fed in the graph through a dictionary as shown
print(sess.run(linear_prediction, {placeholder_node:[[1],[2],[3]]}))
 which gives the outputs as
[[ 0.        ]
 [ 0.30000001]
 [ 0.60000002]]
Our  demonstration MNIST code runs the graph 2000 times which means to iterate the improvement process 2000 times.
for train_step in range(2000):
    batch = mnist.train.next_batch(BATCH_SIZE)
    if train_step % 100 == 0:
        loss_value, _ = sess.run([loss, training], {images: batch[0], true_labels: batch[1]})
        print('Loss at ', str(train_step), ' training step is ', str(loss_value))
Summary:
Answer this to finish up chapter 3.
What are the possible tuning options you feel you have with this level of granular control on ML architectures using TF?

End Notes:

This guide will help you getting started with TensorFlow. However, TF is way richer than what has been covered here. Also, given the fact now Keras will be merged into TF core as a high-level API, it makes more sense than ever to take a look at Keras as well.

In the end, I would say to complete the few liner summaries and post it here as comments. And keep checking this page back to read what others wrote as their summaries. As far as implementing anything in TF is concerned, first, take a close look at the code given in the repo. Then try to code as much as you can without looking it. Of course, do take a look if you can’t remember the stuff at all. Also, don’t hesitate to reach out in case you stumble upon any seemingly unsolvable task or doubts or concerns. I’ve shared all my social media links that I use frequently.

Thank you, everyone, for your time and happy coding!!

Other Resources:
You can find me here as well:

 

Answers:

Quiz 1:
1 – b, 2 – a, 3 – c

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s