CNN in numpy
The purpose of this article is to understand internal calculations of CNN(Convolution Neural Network)
.
Most of ML applications are actively using CNN(Convolution Neural Network). It is quite easy to create a CNN layer thanks to Google Tensorflow
. But, understanding its internal logic from scratch will help you to develop and improve your models.
I found a great illustration of CNN in Stanford CS231n lecture
. So, I decided to use these inputs (7x7x3) and filter weights (3x3x3) as the initial values. Also, I’ll derive the 1-step CNN result using Tensorflow and numpy.
Credit for this image goes to Andrej Karpathy.
1. Setting up the initial values
First, I’ll set the initial values for CNN. We need input values x
and weights w
. The initial values of input values and weights are from the above image of Stanford CS231n lecture
.
import numpy as np
import tensorflow as tf
from tensorflow.python.framework import dtypes
# 1. Setting up initial values
x = np.zeros((7, 7, 3))
x[:, :, 0] = np.mat(
"0 0 0 0 0 0 0;0 0 1 0 1 0 0;0 2 1 0 1 2 0;0 0 2 0 0 1 0;0 2 0 1 0 0 0;0 0 0 1 2 2 0;0 0 0 0 0 0 0"
).A
x[:, :, 1] = np.mat(
"0 0 0 0 0 0 0;0 1 0 0 1 1 0;0 0 2 2 1 1 0;0 2 1 2 1 0 0;0 2 1 1 2 2 0;0 1 2 0 2 2 0;0 0 0 0 0 0 0"
).A
x[:, :, 2] = np.mat(
"0 0 0 0 0 0 0;0 2 1 1 1 1 0;0 2 2 1 2 1 0;0 1 1 0 2 2 0;0 2 1 2 2 0 0;0 1 2 2 0 0 0;0 0 0 0 0 0 0"
).A
x = np.reshape(x, (1, 7, 7, 3))
# print("x:",x)
w = np.zeros((3, 3, 3, 2))
w[:, :, 0, 0] = np.mat("0 0 1;-1 1 1;0 1 0").A
w[:, :, 1, 0] = np.mat("1 1 1;0 1 1;0 1 0").A
w[:, :, 2, 0] = np.mat("-1 0 0;-1 1 1;0 -1 0").A
# w1 = np.zeros((3,3,3))
w[:, :, 0, 1] = np.mat("0 0 0;1 1 -1;-1 1 1").A
w[:, :, 1, 1] = np.mat("0 1 -1;1 1 -1;-1 1 -1").A
w[:, :, 2, 1] = np.mat("1 1 0;-1 -1 0;0 -1 1").A
stride = 2
scope = "conv_in_numpy"
act = tf.nn.relu # activation
pad = 'VALID' # padding
nf = 2 # number of filters
rf = 3 # filter size
b = [1, 0] # bias
np_o = np.zeros((1, 3, 3, 2))
s = stride
2. Convolution in Tensorflow
On this step, I’ll get the 1-step CNN result using Tensorflow.
# 2. CNN in Tensorflow
print("--- Convolution in Tensorflow ---")
tf_x = tf.constant(x, dtype=dtypes.float32)
with tf.Session() as sess:
with tf.variable_scope(scope):
nin = tf_x.get_shape()[3].value
tf_w = tf.get_variable("w", [rf, rf, nin, nf], initializer=tf.constant_initializer(w))
tf_b = tf.get_variable(
"b", [nf],
initializer=tf.constant_initializer(b, dtype=dtypes.float32))
tf_z = tf.nn.conv2d(
tf_x, w, strides=[1, stride, stride, 1], padding=pad) + b
tf_h = act(tf_z)
sess.run(tf.global_variables_initializer())
tf_o = sess.run(tf_z)
print("tf_o0:\n", tf_o[0, :, :, 0])
print("tf_o1:\n", tf_o[0, :, :, 1])
--- Convolution in Tensorflow --- tf_o0: [[ 6. 4. 3.] [ 13. 7. 4.] [ 10. 9. 5.]] tf_o1: [[ -1. -3. 1.] [ 0. 6. 2.] [ 1. -3. 12.]]
3. Convolution in numpy
Okay, it is time to calculate CNN manually. And its code is impressively simple.
# 3. CNN in numpy
print("--- Convolution in numpy ---")
for z in range(nf):
# print("z:", z)
h_range = int((x.shape[2] - rf) / s) + 1 # (W - F + 2P) / S
for _h in range(h_range):
w_range = int((x.shape[1] - rf) / s) + 1 # (W - F + 2P) / S
for _w in range(w_range):
np_o[0, _h, _w, z] = np.sum(
x[0, _h * s:_h * s + rf, _w * s:_w * s + rf, :] *
w[:, :, :, z]) + b[z]
print("np_o0:\n", np_o[0, :, :, 0])
print("np_o1:\n", np_o[0, :, :, 1])
--- Convolution in numpy --- np_o0: [[ 6. 4. 3.] [ 13. 7. 4.] [ 10. 9. 5.]] np_o1: [[ -1. -3. 1.] [ 0. 6. 2.] [ 1. -3. 12.]]
Finally, we can check the results are same each other using the np.testing library.
np.testing.assert_almost_equal(tf_o, np_o)
4. References
- CNN Animation, Andrej Karpathy, Stanford CS231n lecture : https://cs231n.github.io/convolutional-networks/
- (Cover Image) People with OCD know what to do, they just have trouble doing it, Nicole Wetsman, Popular Science : https://www.popsci.com/ocd-understanding-compulsions
5. Full Source Code
Here is the full source code on my gist.