CNN in numpy

The purpose of this article is to understand internal calculations of CNN(Convolution Neural Network).

Most of ML applications are actively using CNN(Convolution Neural Network). It is quite easy to create a CNN layer thanks to Google Tensorflow. But, understanding its internal logic from scratch will help you to develop and improve your models.

I found a great illustration of CNN in Stanford CS231n lecture. So, I decided to use these inputs (7x7x3) and filter weights (3x3x3) as the initial values. Also, I’ll derive the 1-step CNN result using Tensorflow and numpy.

png

Credit for this image goes to Andrej Karpathy.

1. Setting up the initial values

First, I’ll set the initial values for CNN. We need input values x and weights w. The initial values of input values and weights are from the above image of Stanford CS231n lecture.

import numpy as np
import tensorflow as tf
from tensorflow.python.framework import dtypes

# 1. Setting up initial values

x = np.zeros((7, 7, 3))
x[:, :, 0] = np.mat(
    "0 0 0 0 0 0 0;0 0 1 0 1 0 0;0 2 1 0 1 2 0;0 0 2 0 0 1 0;0 2 0 1 0 0 0;0 0 0 1 2 2 0;0 0 0 0 0 0 0"
).A
x[:, :, 1] = np.mat(
    "0 0 0 0 0 0 0;0 1 0 0 1 1 0;0 0 2 2 1 1 0;0 2 1 2 1 0 0;0 2 1 1 2 2 0;0 1 2 0 2 2 0;0 0 0 0 0 0 0"
).A
x[:, :, 2] = np.mat(
    "0 0 0 0 0 0 0;0 2 1 1 1 1 0;0 2 2 1 2 1 0;0 1 1 0 2 2 0;0 2 1 2 2 0 0;0 1 2 2 0 0 0;0 0 0 0 0 0 0"
).A

x = np.reshape(x, (1, 7, 7, 3))
# print("x:",x)

w = np.zeros((3, 3, 3, 2))
w[:, :, 0, 0] = np.mat("0 0 1;-1 1 1;0 1 0").A
w[:, :, 1, 0] = np.mat("1 1 1;0 1 1;0 1 0").A
w[:, :, 2, 0] = np.mat("-1 0 0;-1 1 1;0 -1 0").A

# w1 = np.zeros((3,3,3))
w[:, :, 0, 1] = np.mat("0 0 0;1 1 -1;-1 1 1").A
w[:, :, 1, 1] = np.mat("0 1 -1;1 1 -1;-1 1 -1").A
w[:, :, 2, 1] = np.mat("1 1 0;-1 -1 0;0 -1 1").A

stride = 2
scope = "conv_in_numpy"
act = tf.nn.relu  # activation
pad = 'VALID'  # padding
nf = 2  # number of filters
rf = 3  # filter size
b = [1, 0]  # bias

np_o = np.zeros((1, 3, 3, 2))
s = stride

2. Convolution in Tensorflow

On this step, I’ll get the 1-step CNN result using Tensorflow.

# 2. CNN in Tensorflow

print("--- Convolution in Tensorflow ---")

tf_x = tf.constant(x, dtype=dtypes.float32)

with tf.Session() as sess:
    with tf.variable_scope(scope):
        nin = tf_x.get_shape()[3].value
        tf_w = tf.get_variable("w", [rf, rf, nin, nf], initializer=tf.constant_initializer(w))
        tf_b = tf.get_variable(
            "b", [nf],
            initializer=tf.constant_initializer(b, dtype=dtypes.float32))
        tf_z = tf.nn.conv2d(
            tf_x, w, strides=[1, stride, stride, 1], padding=pad) + b
        tf_h = act(tf_z)
        sess.run(tf.global_variables_initializer())
        tf_o = sess.run(tf_z)
        print("tf_o0:\n", tf_o[0, :, :, 0])
        print("tf_o1:\n", tf_o[0, :, :, 1])

--- Convolution in Tensorflow ---
tf_o0:
 [[  6.   4.   3.]
 [ 13.   7.   4.]
 [ 10.   9.   5.]]
tf_o1:
 [[ -1.  -3.   1.]
 [  0.   6.   2.]
 [  1.  -3.  12.]]

3. Convolution in numpy

Okay, it is time to calculate CNN manually. And its code is impressively simple.

# 3. CNN in numpy

print("--- Convolution in numpy ---")

for z in range(nf):
    # print("z:", z)
    h_range = int((x.shape[2] - rf) / s) + 1  # (W - F + 2P) / S
    for _h in range(h_range):
        w_range = int((x.shape[1] - rf) / s) + 1  # (W - F + 2P) / S
        for _w in range(w_range):
            np_o[0, _h, _w, z] = np.sum(
                x[0, _h * s:_h * s + rf, _w * s:_w * s + rf, :] *
                w[:, :, :, z]) + b[z]

print("np_o0:\n", np_o[0, :, :, 0])
print("np_o1:\n", np_o[0, :, :, 1])

--- Convolution in numpy ---
np_o0:
 [[  6.   4.   3.]
 [ 13.   7.   4.]
 [ 10.   9.   5.]]
np_o1:
 [[ -1.  -3.   1.]
 [  0.   6.   2.]
 [  1.  -3.  12.]]

Finally, we can check the results are same each other using the np.testing library.

np.testing.assert_almost_equal(tf_o, np_o)

4. References

CNN Animation, Andrej Karpathy, Stanford CS231n lecture : https://cs231n.github.io/convolutional-networks/
(Cover Image) People with OCD know what to do, they just have trouble doing it, Nicole Wetsman, Popular Science : https://www.popsci.com/ocd-understanding-compulsions

5. Full Source Code

Here is the full source code on my gist.