Others

Effects of hyperparameters on mlp’s

This week I was doing some simple experiments with MLP, trying out different architecture and on different learning rates with random batch normalization and sometimes droput. There was no such definite goal of this experiment except to observe the behavior of Neural networks when we change learning rates on different architectures.

The goal of this article is to share some of simple findings from my experiment, there is no sequence for these test and are completely based on intuition.I have only used keras as to make implementation much easier.

First architecture

This was first architecture so I decided not use any batch normalization or dropouts. The architecture was quite simple lets look at implementation,

---

import keras
from keras.datasets import mnist
import keras.backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout , BatchNormalization

(x_train,y_train), (x_test,y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0],784)
x_test = x_test.reshape(x_test.shape[0],784)

y_train = keras.utils.to_categorical(y_train,num_classes=10)
y_test = keras.utils.to_categorical(y_test, num_classes=10)

n_hidden_1 = 10
n_hidden_2 = 7
n_hidden_3 = 3

model = Sequential()
model.add(Dense(784,input_shape=(784,),activation='relu'))
model.add(Dense(n_hidden_1,activation='relu'))
model.add(Dense(n_hidden_2,activation='relu'))
model.add(Dense(n_hidden_3,activation='relu'))
model.add(Dense(10,activation='softmax'))

model.compile(loss = keras.losses.categorical_crossentropy,optimizer=keras.optimizers.Adam(lr=0.01),metrics=['accuracy'])
model.fit(x_train,y_train,epochs=4)

score = model.evaluate(x_test,y_test)

---

This is beauty of keras the code itself is self explanatory. The output of this code,

---

Epoch 1/10
60000/60000 [==============================] - 5s 87us/step - loss: 2.3320 - acc: 0.1110
Epoch 2/10
60000/60000 [==============================] - 5s 79us/step - loss: 2.3024 - acc: 0.1099
Epoch 3/10
60000/60000 [==============================] - 5s 80us/step - loss: 2.3024 - acc: 0.1095
Epoch 4/10
60000/60000 [==============================] - 5s 81us/step - loss: 2.3024 - acc: 0.1104

---

As you can see after the loss decrease from 2.33 to 2.30 it remains constant and did not further decreased, I don’t find any mathematical reason apart from this one due to the fact we have taken learning rate quite high take the optimizer either at some saddle point or a local minima from where it cannot move in any direction further. The results were quite similar while using SGD also which is a poor optimizer than adam.

---

Epoch 1/10
60000/60000 [==============================] - 5s 79us/step - loss: 2.3052 - acc: 0.1119
Epoch 2/10
60000/60000 [==============================] - 4s 74us/step - loss: 2.3013 - acc: 0.1124
Epoch 3/10
60000/60000 [==============================] - 4s 74us/step - loss: 2.3013 - acc: 0.1124
Epoch 4/10
60000/60000 [==============================] - 5s 77us/step - loss: 2.3013 - acc: 0.1124

---

Second Architecture

In this i have tried something unusual, rather than having small first hidden layer and increasing the number of neurons  for further hidden layers,

code below,

---

import keras
from keras.datasets import mnist
import keras.backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout , BatchNormalization

(x_train,y_train), (x_test,y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0],784)
x_test = x_test.reshape(x_test.shape[0],784)

y_train = keras.utils.to_categorical(y_train,num_classes=10)
y_test = keras.utils.to_categorical(y_test, num_classes=10)

n_hidden_1 = 32
n_hidden_2 = 64
n_hidden_3 = 128

model = Sequential()
model.add(Dense(784,input_shape=(784,),activation='relu'))
model.add(Dense(n_hidden_1,activation='relu'))
model.add(Dense(n_hidden_2,activation='relu'))
model.add(Dense(n_hidden_3,activation='relu'))
model.add(Dense(10,activation='softmax'))

model.compile(loss = keras.losses.categorical_crossentropy,optimizer=keras.optimizers.SGD(lr=0.01),metrics=['accuracy'])
model.fit(x_train,y_train,epochs=10)

score = model.evaluate(x_test,y_test)

---

output,

---

Epoch 1/10
60000/60000 [==============================] - 6s 96us/step - loss: 14.4698 - acc: 0.1022
Epoch 2/10
60000/60000 [==============================] - 5s 89us/step - loss: 14.4711 - acc: 0.1022
Epoch 3/10
60000/60000 [==============================] - 5s 88us/step - loss: 14.4711 - acc: 0.1022
Epoch 4/10
60000/60000 [==============================] - 5s 90us/step - loss: 14.4711 - acc: 0.1022

---

This output was not surprising even the cost increased rather than decreasing couple of reason for this,

• Learning rate is high so it might stuck at some local minima  or saddle point
• The reason for why cost increased is because of not just one,
  • Sgd don’t take all the points in consideration to find the next best direction
  • hidden layers neurons are responsible for finding hidden patterns in data so less number of neurons means less number of hidden patters recognized and due to the fact subsequent layers findings depends upon previous layers so if not including large number of neurons in first layers affects performance but this might not be the possible explanation for above result.

Third Architecture,

In this i have only used 2 layers with same number of neurons in it with dropout of 0.5 and learning rate 0.0001

---

import keras
from keras.datasets import mnist
import keras.backend as K
from keras.models import Sequential
from keras.layers import Dense, Dropout , BatchNormalization

(x_train,y_train), (x_test,y_test) = mnist.load_data()
x_train = x_train.reshape(x_train.shape[0],784)
x_test = x_test.reshape(x_test.shape[0],784)

y_train = keras.utils.to_categorical(y_train,num_classes=10)
y_test = keras.utils.to_categorical(y_test, num_classes=10)

n_hidden_1 = 10
n_hidden_2 = 10


model = Sequential()
model.add(Dense(784,input_shape=(784,),activation='relu'))
model.add(Dense(n_hidden_1,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(n_hidden_2,activation='relu'))
model.add(Dense(10,activation='softmax'))

model.compile(loss = keras.losses.categorical_crossentropy,optimizer=keras.optimizers.SGD(lr=0.0001),metrics=['accuracy'])
model.fit(x_train,y_train,epochs=6)

---

the output of this is quite what expected,

---

Epoch 1/10
60000/60000 [==============================] - 22s 366us/step - loss: 2.4244 - acc: 0.1302
Epoch 2/10
60000/60000 [==============================] - 22s 361us/step - loss: 2.1681 - acc: 0.1766
Epoch 3/10
60000/60000 [==============================] - 22s 364us/step - loss: 2.1075 - acc: 0.2006
Epoch 4/10
60000/60000 [==============================] - 22s 361us/step - loss: 2.0722 - acc: 0.2170
Epoch 5/10
60000/60000 [==============================] - 21s 358us/step - loss: 2.0553 - acc: 0.2234
Epoch 6/10
60000/60000 [==============================] - 22s 359us/step - loss: 2.0408 - acc: 0.2253

---

The loss is decreasing with each epoch as the learning rate is low so the network is learning and not skipping global minima(just an observation). This work quite similar with adam.

---

Epoch 1/10
60000/60000 [==============================] - 41s 679us/step - loss: 2.4292 - acc: 0.1399
Epoch 2/10
60000/60000 [==============================] - 40s 660us/step - loss: 2.2645 - acc: 0.1314
Epoch 3/10
60000/60000 [==============================] - 40s 662us/step - loss: 2.2330 - acc: 0.1459
Epoch 4/10
60000/60000 [==============================] - 40s 663us/step - loss: 2.1963 - acc: 0.1633
Epoch 5/10
60000/60000 [==============================] - 40s 671us/step - loss: 2.1695 - acc: 0.1762
Epoch 6/10
60000/60000 [==============================] - 40s 664us/step - loss: 2.1702 - acc: 0.1813

---

The observations from above experiment

• A good learning rate is in between .001 – .00001.
• Use Adam for quite stable results.
• size of first hidden layer should be large and decrease it as the network grows deeper
• Use relu to avoid problems of diminishing gradients in deep mlp’s.
• Must use dropouts to avoid over fitting.

Confidence Interval

What is Confidence interval ?

“Confidence intervals tells about the certainty of  estimations by providing bounds around it.”

Feature selection

“How good your machine learning model predict depends upon what data you use to train it.”

If the data is itself bad don’t expect good result, now the immediate question arises “How you define bad data ?”, the answer of this question is quite difficult as the quality of data depends upon many factors but among all factors the choice of features used to create data is most important. The choice of the features must be align with objective of the problem. Choosing features is not a type of task where one should only depend upon intuition. Feature selection is a statistical technique which used to find relevant features.

The goal of this article is to answer following questions.

• What is feature selection ?
• What are filter methods for feature selection ?
• What are wrapper methods for feature selection ?
• What are embedded methods for feature selection ?

What is feature selection ?

Feature selection refers to attribute  or variable selection for a problem. This must be performed before feeding data to an algorithm because of following reasons,

• Higher the dimension of the data slower the process of training model.
• Some features may be completely irrelevant to predict output.
• It affects the accuracy of model .
• In some cases overfitting can also be avoided.

What are filter methods for feature selection ?

Filter selection methods involves methods in which we use some statistical tests in order to find the relevance of the features with respect to output. These tests are performed without considering the algorithm of the model. The features which are  selected, are purely dependent upon their statistical test scores.

Some of the most popular statistical tests performed are,

• Co-variance : Measuring co-variance by taking a subset of features is often a good, as by doing this we can get a fairly good idea as how are features behaves with respect to each other.

• Pearson correlation:  Though the co-variance provides good information about correlation between variables but the disadvantage with using Co-variance is there is no upper-bound on co-variance score this makes it little difficult to judge how much the features are aligned with each other for that we use Pearson correlation.

• LDA : Linear discriminant analysis is one of the techniques where we try to select features which will be based as how good they separates one of two classes.

 

What are wrapper methods for feature selection ?

The feature selection techniques that comes under this heading, selects features which depends upon what type of algorithm you use for your model. The idea is simple select a single feature or a subset of feature and feed it into model and store accuracy, after certain number of combinations which ever subset performs the best choose it.

The techniques which comes under this heading is,

• Forward feature selection : The idea is simple first choose a single feature which gives the best performance then keep adding features on top of it and checks the performance after every addition and stop this process if the performance does not improve or improve slightly which is not very significant.
• Backward Elimination : The idea of this technique is quite opposite as that of forward feature selection, in this technique we first start with  all features and start eliminating features one by one till there is no significant decrement in performance of the model.
• Recursive selection: This is not a very computationally friendly technique as it first select select best features and worst features at each stage and then find the best features from left over and checks the performance, it does this process recursively. Then after all features are finished it ranks the features according to which they are eliminated. Then the algorithm chooses top ranks features.

There is no all in one package which implements all these techniques but there are some packages which can help sklearn, boruta etc, for rest of techniques implement it by yourself.

Embedded feature selection

These methods of feature selection are built on top of filter and wrapper feature selection. One of the most popular example of embedded feature selection is regularization. They often provides constraint over the objective function which avoids the problem of coefficients of the weights becoming zero thus helps overfitting of the model. These methods are completely algorithm dependent.

 

Weight Initialization in neural network

Weight initialization is one of the major factors that directly affects the convergence of the model i.e how accurately the optimizer minimize the loss function. Before we even get started I am assuming that the reader are aware, as what are neural networks, what are hidden layers, how they are connected etc. Though I will provide a abstract overview  of working of a neural network. We will be only considering MLP architecture.

The basic workflow of neural network is,

1. Initialize all the weights and biases of the network
2. Forward pass
3. Compute the loss
4. Calculate the gradients of the loss function
5. backpropagate the loss
6. update the weights.

Before we even discuss techniques to initialize the weights let’s first discuss what happens if we don’t initialize the weights properly.

1. Initialize all weights to zero : This is one of the worst mistake you can possibly do while initializing as doing this will make your neural network work only as a linear model rather than a non-linear as making all the weights as 0 means only the biases will be responsible for the direction of the decision boundary.
2. Very small values of weights with sigmoid activation : This is one of the cause of vanishing gradients problem i.e the weights itself are so small that makes the gradients almost negligible makes it difficult for the network to learn something and eventually stopping the minimization of loss function.
3. Exploding gradients problem : In this situation the value of the gradients itself becomes so large that the model never converges to global minima as it keeps oscillating over global minima or sometimes instead of converging the direction of the gradients is away from the minima making it impossible to to reach there.

Understanding the need of smart initializing is important as then after you will know by not doing smart initializing of weights can cause serious problems.

Techniques of weight initialization

Random initializing:

This one is most commonly followed by beginners as generating random numbers from a normal distribution typically with mean equals 0 and stadndard-dev 0.2 works well in most cases and in-fact in most cases you will not face any problem except that by using this technique some network might converges slowly . The weights can following this technique can be initialize quite easily using just numpy.

---

weights = sigma *  np.random.randn(Size_L,Size_L-1) * mean

---

the downside of using this technique is that if the value of mean and sigma is not chosen wisely can sometime cause the problem of vanishing gradients or exploding gradients.

Initializing weights keeping relu activation in mind:

Initializing weight keeping what activation you are using is a good start as what non-linear activation you use is greatly influenced by the weights you initialize.

For relu you can use weights coming from normal distribution with mean of 1 and variance of sqrt(2/size of previous hidden layer).

---

sigma = np.sqrt(2/Size_L-1)
weights = sigma *  np.random.randn(Size_L,Size_L-1)

---

If using tanh activation:

In case of tanh you can use Xavier initialization, this is similar to above initialization except in place of 2 it is 1 in xavier.

---

sigma = np.sqrt(1/Size_L-1)
weights = sigma *  np.random.randn(Size_L,Size_L-1)

---

Another initialization which is modification of xavier is,

---

sigma = np.sqrt(2/Size_L-1+Size_L)
weights = sigma *  np.random.randn(Size_L,Size_L-1)

---

These initialization keeps the weights in range neither very small nor big, thus it avoid the possibility of slow convergence and due to the fact the are not big enough to create the problem of exploding gradients.

How to learn Mathematics for Machine learning

Learning and understanding mathematics is key to understand machine learning algorithms, as machine learning algorithms is applied mathematics. Mathematics is just a language as all other language but it is so unambiguous and precise than we could explain phenomenon using it. Understanding Mathematics is very crucial for machine learning, and by understanding it means what each step means both theoretically and geometrically (not applicable in all branch of mathematics).

The growth in the field of machine learning in last decade has become lightning fast due to many reason,

• Available of datasets
• Hardware to support the computation (one of the most important reason)
• Exceptional results using deep learning

The sub field of machine learning called deep learning is heavily based on mathematics and without understanding mathematics you will only be a script kiddie in this field. The discussion does not ends here, as if the field is growing fast then in that case your learning should also be fast but most importantly clear.

In this article I will discuss the methods I use to learn mathematics for machine learning fast and effective, and before even starting you must keep this in mind that anybody can learn this field you don’t have to be some super genius, all you need is thrive to learn and most importantly consistency to do the things daily even if you don’t like doing it on some days.

There are specifically four branches of mathematics that are used in machine learning,

1. Linear algebra
2. Probability and statistics
3. calculus
4. information theory

though discussing what to learn is not the goal of this article but for the sake of information I have provided.

Identify what to learn

This is a very essential step of learning as before even start what to learn you must know what not to learn as for any beginner the mathematics involve in machine learning can somewhat be over whelming at first, so that means you must understand this, you cannot learn all things at once no one can, as learning is not a goal it is a process itself. There are couple of things you can workout to eliminate options in front of you,

1. Pick a goal like learning a particular topic or building a minor project, it will be very beneficial if you choose some minor project as your goal. After setting up  the goal  find out  all the mathematics involve in it and only learn up till that and don’t get distracted in between.
2. If you find any topic which is based on the understanding of many other topics then try to understand it in abstract manner at first.

Using above tricks you will eliminate many options and learn only what you need to take your game to next level.

How you learn

This is more of self observation, some people understand better using audio, video some with written text and some are comfortable with both, and most important point is no method is better than other as long you are learning consistently. If I have less time than i prefer watching lectures on topics at 2x or 3x but if I had time then I prefer to go through some standard books on particular topics which will provide very in-depth knowledge. You must observe yourself as what type of learning you are most comfortable with rather than trying hard to learn in some other manner as if you are beginner the most important thing is to learn not what is the way of learning you use. Though it is recommended you should develop habit  of reading, specifically equations as it will help you to read research paper which is essential to keep yourself updated with this field.

Choose the right option

Choosing the right option of the place from where you will learn, is very crucial. If the material you have chosen for your learning is not good for you, then chances are you will quit before even starting.There is no such thing as good or bad for learning, it all depends which is good for you, The main point you need to consider is try breath-first approach rather than depth first, try out different options present on internet and find out which one suits your frequency. I would recommend rather than approach math first before even taking any machine learning course is not a good idea as after taking a machine learning course it will take you to a place from where you can select what to learn and even compare options you have.

Balancing between depth-first and breath-first

Going too deep in one topic or learning every topic at only abstract level are both not a good situation to live in, you must find a right balance between these two trade-off, what I would suggest is go deep only till at the point you understand every bit of your project as why things are happening. If you successfully arrived at this point then its time to go through next topic as for any beginner going too deep in one topic will make you consume your time rather making you learn to build something out of it.

Bonus tip

Learning mathematics can be sometime over whelming or boring so it is essential to keep two things in mind,

1. if possible try to understand things in geometric manner as it will help you to build strong understanding about concepts.
2. Never hesitate to look out for help, I know this might be difficult for some people, but the reality is no one can know it all we all learn by helping each other.

Introduction to tensorflow Part 3

If you are new to tensorflow, we recommend to start this series from part 1

This is part 3 of tensorflow series. The goal of every single article is to make you understand one of the most popular deep learning library out there. This series is entirely focused on beginners, who are either starting to learn deep learning and want to built their own neural nets or want to build state-of-the-art neural network’s in tensorflow. This series of article will specifically covers basics of tensorflow for python, there is another version of tensorflow.js for javascript developers but that will be covered in next series.In last part of the series we will be building a MLP  for MNIST dataset using tensorflow.

After reading this article you will be,

• Able to design MLP
• Implement it using tensorflow

First check if tensorflow is working

This can be done just by importing tensorflow,

---

import tensorflow as tf

---

if every thing is correct this should be execute without giving any error, but there is a possibility that ipython displays warning for some library specific functions.

Check for computing devices like cpu or gpu

---

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

---

if you have only cpu it will display “/device:CPU:0” and with gpu  “/gpu:0” followed by gpu name,
note : Tensorflow only works with nvidia gpu’s as it currently only uses Cuda library.

The MLP architecture which we will be building will contain,

1. Input layer of size 784, as each image size is 28×28 and we are using flatten version of it
2. hidden layer 1 : size 10, i.e mlp 1st hidden layer will contain 10 neuron
3. hidden layer 2 : size 5 i.e mlp 2nd hidden layer will contain 5 neuron

The size of the layer are the hyperparameters of the network, which have been set by using some experience, don’t worry too much about hyperparameter tuning.

let’s start building a mlp

import data

---

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

---

Mnist data is already present in sample datasets of tensorflow, the following code will download mnist dataset and place it inside “MNIST_data” folder.

Setting up hyperparameters

---

input_layer = 784
hidden_layer_1 = 10
hidden_layer_2 = 5
output_layer= 10

---

These are just variables which we will be using for storing layer configuration of our network.

Initialize placeholders for input layer and output layer

---

x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])

---

you can view these placeholders as describing size of input and output of the network.

Initialize weights of the model

---

weights = {
'h1': tf.Variable(tf.random_normal([input_layer, hidden_layer_1],stddev=0.062, mean=0)), 
'h2': tf.Variable(tf.random_normal([hidden__layer_1,hidden_layer_2],stddev=0.125, mean=0)), 
'out': tf.Variable(tf.random_normal([hidden_layer_2, output_layer],stddev=0.120, mean=0)) 
}

---

We have used a python dictionary to store all our weights layer-by-layer, I find this a very elegant approach rather than using individual variables for each layer.

Initializing biases

---

biases = {
'b1': tf.Variable(tf.random_normal([hidden_layer_1])), 
'b2': tf.Variable(tf.random_normal([hidden_layer_2])), 
'out': tf.Variable(tf.random_normal([output_layer])) 
}

---

We have followed the same approach for biases also.

Model parameters

---

epochs = 15 #total no of times the whole dataset will be used
learning_rate = 0.001 #learning rate of the algorithm
batch_size = 100 #batch size in case of optimizers other that standard gradient descent.

---

These parameter of the model will be fixed during whole computation

Defining MLP

This is the core step of this whole article.Before reading further if you are unfamiliar with activation function click here.

---

def MLP(x, weights, biases):

    #first hidden layer with relu activation
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)

    #second layer with relu activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2']) 
    layer_2 = tf.nn.relu(layer_2)

    #output layer with sigmoid activation
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out'] 
    out_layer = tf.nn.sigmoid(out_layer)


return out_layer

---

If you walk through carefully , you will find that the procedure is quite simple as all we are doing is

“Weight times input add a bias, activate “

Defining our predicted output

---

y_predicted  = MLP(x, weights, biases)

---

y_predicted is the output produced by MLP.

Defining our loss function

we will be using one of the famous loss function the log loss or cross-entropy but as our task is one-vs-all classification we will be using softmax log loss.

---

loss_function = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = y_predicted, labels = y))

---

the whole idea is simple, just take the mean of all softmax results.

Defining our optimization algorithm

The most popular choices among all optimizers are Adadelta or adam but for the simplicity we will be using stochastic gradient descent

---

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss_function)

---

Let’s run our model

This step will make use of all the steps mentioned above.

---

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    for epoch in range(epochs):
        train_avg_cost = 0.
        test_avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)

            for i in range(total_batch): 
                x_train, y_train = mnist.train.next_batch(batch_size)

                cost = sess.run(loss_function, feed_dict={x: mnist.test.images, y: mnist.test.labels})
                test_avg_cost += c / total_batch

            print("Epoch:",epoch+1, "train cost",train_avg_cost, "test cost ",test_avg_cost)

---

The pseudocode of above tensorflow code is,

1. Begin the  tensorflow session
2. Initialize all the above mentioned variables
3. for all epochs
   1. initialize avg cost of trainining for each epoch
   2. initialize avg cost of test for each epoch
   3. initialize the total number of batch, just divide whole training data with batch size
   4. for each batch in one epoch
      1. train the network over train each batch in epoch and compute training data cost and test data cost
   5. print epoch no , train cost and test cost
4. end

Congratulation you have just implemented your first tensorflow mlp model which will be able to recognize handwritten digits. In next article we will try to cover a simple implementation of convnet for Mnist dataset.

For part 4 click here.