{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "<tr>    \n",
    "<td>\n",
    "<img src=\"http://home.agh.edu.pl/~horzyk/im/AdrianHorzyk140h.png\" alt=\"Adrian Horzyk, Professor\" title=\"Adrian Horzyk, Professor\" />        \n",
    "</td> \n",
    "<td style=\"text-align: center\">\n",
    "<h1>Visualize and Inspect Representation of Features in ConvNets</h1>\n",
    "<h2><a href=\"http://home.agh.edu.pl/~horzyk/index-eng.php\">Adrian Horzyk</a></h2>\n",
    "<h3>Welcome to the interactive lecture where you can check everything by yourself and experiment!</h3>\n",
    "</td> \n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualizing what convnets learn\n",
    "\n",
    "It is often said that deep learning models are \"black boxes\", learning representations that are difficult to extract and present in a human-readable form. While this is partially true for certain types of deep learning models, it is definitely not true for convnets. The representations learned by convnets are highly amenable to visualization, in large part because they are _representations of visual concepts_. Since 2013, a wide array of techniques have been developed for visualizing and interpreting these representations. The visualized representations of training data (filters) help us to find some adaptation and generalization problems, so it is essential to use visualization to inspect the trained models to find the possible ways for improvement. We won't survey all of them, but we will cover three of the most accessible and useful ones:\n",
    "\n",
    "<ol>\n",
    "    <li><b>Visualizing intermediate convnet outputs (\"intermediate activations\").</b> This is useful to understand how successive convnet layers transform their input, and to get a first idea of the meaning of individual convnet filters.</li>\n",
    "    <li><b>Visualizing convnets filters.</b> This is useful to understand precisely what visual pattern or concept each filter in a convnet is receptive to.</li>\n",
    "    <li><b>Visualizing heatmaps of class activation in an image.</b> This is useful to understand which part of an image where identified as belonging to a given class, and thus allows to localize objects in images.</li>\n",
    "</ol>\n",
    "\n",
    "To demonstrate them, we will use the cat vs. dog classification problem with the convnets trained from scratch or using VGG-16 network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.3.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Visualizing intermediate activations\n",
    "\n",
    "Visualizing intermediate activations consists in displaying the feature maps that are output by various convolution and pooling layers in a network for given a certain input. The output of a layer is often called its <b>activation</b>, i.e., the output of the activation function. This gives a view into how an input is decomposed unto the different filters learned by the network. These feature maps we want to visualize have 3 dimensions: width, height, and depth (channels). Each channel encodes relatively independent features, so the proper way to visualize these feature maps is by independently plotting the contents of every channel, as a 2D image.\n",
    "\n",
    "Let's start by loading model 2 that we saved previously. You can use the model stored by me or change the path and use the model from your experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_5 (Conv2D)            (None, 148, 148, 32)      896       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 74, 74, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 72, 72, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_6 (MaxPooling2 (None, 36, 36, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 34, 34, 128)       73856     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_7 (MaxPooling2 (None, 17, 17, 128)       0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 15, 15, 128)       147584    \n",
      "_________________________________________________________________\n",
      "max_pooling2d_8 (MaxPooling2 (None, 7, 7, 128)         0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 6272)              0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 6272)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 512)               3211776   \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 513       \n",
      "=================================================================\n",
      "Total params: 3,453,121\n",
      "Trainable params: 3,453,121\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "# The directory where we will store our smaller dataset\n",
    "# It can be downloaded from 'http://horzyk.pl/ml/data/kaggle/processed/cats_and_dogs_small.rar'\n",
    "base_dir = 'C:/Users/Adria/ml/data/kaggle/processed/cats_and_dogs_small'\n",
    "\n",
    "# The directory where we will store our models\n",
    "# It can be downloaded from 'http://horzyk.pl/ml/models/'\n",
    "models_dir = 'C:/Users/Adria/ml/models/'\n",
    "\n",
    "model = load_model(models_dir + 'cats_and_dogs_small_2.h5')\n",
    "\n",
    "model.summary()  # Show a reminder of this model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will be the input image we will use -- a picture of a cat, not part of images that the network was trained on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 150, 150, 3)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# It can be downloaded from 'http://horzyk.pl/ml/data/kaggle/processed/cats_and_dogs_small/test/cats/cat.1700.jpg'\n",
    "img_path = base_dir + '/test/cats/cat.1700.jpg'\n",
    "\n",
    "# We preprocess the image into a 4D tensor\n",
    "from keras.preprocessing import image\n",
    "\n",
    "img = image.load_img(img_path, target_size=(150, 150))\n",
    "img_tensor = image.img_to_array(img)\n",
    "img_tensor = np.expand_dims(img_tensor, axis=0)\n",
    "# Remember that the model was trained on inputs that were preprocessed in the following way, so we must do the same now:\n",
    "img_tensor /= 255.\n",
    "\n",
    "# Its shape is (1, 150, 150, 3)\n",
    "print(img_tensor.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does our loaded image look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.imshow(img_tensor[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to extract the feature maps, we will create a Keras model that takes batches of images as input, and outputs \n",
    "the activations of all convolutional and pooling layers. To do this, we will use the Keras class `Model` that is instantiated using two arguments: an input tensor (or a list of input tensors), and an output tensor (or a list of output tensors). The resulting class is a Keras model, just like the `Sequential` models, mapping the specified inputs to the specified outputs. What sets the `Model` class apart is that it allows for models with __multiple outputs__, unlike `Sequential`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import models\n",
    "\n",
    "# Extracts the outputs of the top 8 layers:\n",
    "layer_outputs = [layer.output for layer in model.layers[:8]]\n",
    "\n",
    "# Creates a model that will return these outputs, given the model input:\n",
    "activation_model = models.Model(inputs=model.input, outputs=layer_outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When fed an image input, this model returns the values of the layer activations (outputs of layers) in the original model. This might be the first time you encounter <b>a multi-output model</b>. Until now the models we worked with only had exactly one input and one output. In the general case, a model could have any number of inputs and outputs. This one has one input and 8 outputs, one output per layer activation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will return a list of 5 Numpy arrays:\n",
    "# one array per layer activation\n",
    "activations = activation_model.predict(img_tensor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, this is the activation of the first convolution layer for our cat image input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 148, 148, 32)\n"
     ]
    }
   ],
   "source": [
    "first_layer_activation = activations[0]\n",
    "print(first_layer_activation.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's a 148x148 feature map with 32 channels. Let's try visualizing three sample channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 59444 characters]\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 81844 characters]\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 82956 characters]\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.matshow(first_layer_activation[0, :, :, 0], cmap='viridis')\n",
    "plt.show()\n",
    "plt.matshow(first_layer_activation[0, :, :, 7], cmap='viridis')\n",
    "plt.show()\n",
    "plt.matshow(first_layer_activation[0, :, :, 12], cmap='viridis')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, these channels encode and detect different features (presented as bright points)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot a complete visualization of all the activations in the network. We'll extract and plot every channel in each of our 8 activation maps, and we will stack the results in one big image tensor, with channels stacked side by side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Adria\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:27: RuntimeWarning: invalid value encountered in true_divide\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 228560 characters]\n",
      "text/plain": [
       "<Figure size 1152x144 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 237676 characters]\n",
      "text/plain": [
       "<Figure size 1152x144 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 302720 characters]\n",
      "text/plain": [
       "<Figure size 1152x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 188044 characters]\n",
      "text/plain": [
       "<Figure size 1152x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 267516 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 109196 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 55200 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 26176 characters]\n",
      "text/plain": [
       "<Figure size 1152x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# These are the names of the layers, so can have them as part of our plot\n",
    "layer_names = []\n",
    "for layer in model.layers[:8]:\n",
    "    layer_names.append(layer.name)\n",
    "\n",
    "images_per_row = 16\n",
    "\n",
    "# Now let's display our feature maps\n",
    "for layer_name, layer_activation in zip(layer_names, activations):\n",
    "    # This is the number of features (channels) in the feature map\n",
    "    n_features = layer_activation.shape[-1]\n",
    "\n",
    "    # The feature map has shape (1, size, size, n_features)\n",
    "    size = layer_activation.shape[1]\n",
    "\n",
    "    # We will tile the activation channels in this matrix\n",
    "    n_cols = n_features // images_per_row\n",
    "    display_grid = np.zeros((size * n_cols, size * images_per_row))\n",
    "\n",
    "    # We'll tile each filter into this big horizontal grid\n",
    "    for col in range(n_cols):\n",
    "        for row in range(images_per_row):\n",
    "            channel_image = layer_activation[0, :, :, col * images_per_row + row]\n",
    "            \n",
    "            # Post-process the feature to make it visually palatable\n",
    "            channel_image -= channel_image.mean()\n",
    "            channel_image /= channel_image.std()\n",
    "            channel_image *= 64\n",
    "            channel_image += 128\n",
    "            channel_image = np.clip(channel_image, 0, 255).astype('uint8')\n",
    "            display_grid[col * size : (col + 1) * size, row * size : (row + 1) * size] = channel_image\n",
    "\n",
    "    # Display the grid\n",
    "    scale = 1. / size\n",
    "    plt.figure(figsize=(scale * display_grid.shape[1], scale * display_grid.shape[0]))\n",
    "    plt.title(layer_name)\n",
    "    plt.grid(False)\n",
    "    plt.imshow(display_grid, aspect='auto', cmap='viridis')\n",
    "    \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remarks:\n",
    "\n",
    "* The first layer acts as a collection of various edge detectors. At that stage, the activations are still retaining almost all of the information present in the initial picture.\n",
    "* As we go higher-up, the activations become increasingly abstract and less visually interpretable. They start encoding higher-level concepts such as \"cat ear\" or \"cat eye\". Higher-up presentations carry increasingly less information about the visual contents of the image, and increasingly more information related to the class of the image.\n",
    "* The sparsity of the activations is increasing with the depth of the layer: in the first layer, almost all filters are activated by the input image, but in the following layers more and more filters are usually blank (black rectangles). This means that the pattern encoded by the filter isn't found in the input image.\n",
    "* We can notice that sometimes some filters are very similar to each other; as a result, they code similar features. This might limit the discrimination properties of the network, so maybe the layers should be unfrozen to train better representation and discrimination of the dataset features.\n",
    "* Sometimes the bright pixels are presented outside the body of the cat. It means that the classification of the cats is processed based on the environment features where the cats were usually photographed! Sometimes it might be an advantage, but sometimes it might be a problem, so bear in mind to pay attention to such bright features outside the classified objects.\n",
    "* The features extracted by a layer get increasingly abstract with the depth of the layer. The activations of layers higher-up carry less and less information about the specific input (the image), and more and more information about the target (the class of the image: cat or dog in our case). A deep neural network effectively acts as an __information distillation pipeline__, with raw data going in (RBG images in our case), and getting repeatedly transformed so that irrelevant information gets filtered out (e.g. the specific visual appearance of the image) while useful information get magnified and refined (e.g. the class specifics of the image).\n",
    "\n",
    "The presented process is analogous to the way humans and animals perceive the world: after observing a scene for a few seconds, a human can remember which abstract objects were present in it (e.g. bicycle, tree) but could not retain the specific appearance of these objects. In fact, if you would try to draw a generic bicycle from your mind right now, chances are you could not get it even remotely right, even though you have seen thousands of bicycles in your lifetime. Try it right now: this effect is absolutely real. Your brain has learned to completely abstract its visual input, to transform it into high-level visual concepts while completely filtering out irrelevant visual details, making it tremendously difficult to remember how things around us actually look like."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Visualizing convnet filters\n",
    "\n",
    "\n",
    "Another essential thing about how to inspect the filters learned by a convnet is to display the visual pattern that each filter is meant to respond to. This can be done with __gradient ascent in input space__: applying __gradient descent__ to the value of the input image of a convnet so as to maximize the response of a specific filter, starting from a blank input image. The resulting input image would be one that the chosen filter is maximally responsive to.\n",
    "\n",
    "To do it, we will build a loss function that maximizes the value of a given filter in a given convolution layer; then we will use stochastic gradient descent to adjust the values of the input image so as to maximize this activation value.\n",
    "\n",
    "For instance, here's a loss for the activation of filter 0 in the layer \"block3_conv1\" of the VGG16 network, pre-trained on ImageNet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.applications import VGG16\n",
    "from keras import backend as K\n",
    "\n",
    "model = VGG16(weights='imagenet', include_top=False)\n",
    "\n",
    "layer_name = 'block3_conv1'\n",
    "filter_index = 0\n",
    "\n",
    "layer_output = model.get_layer(layer_name).output\n",
    "loss = K.mean(layer_output[:, :, :, filter_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To implement gradient descent, we will need the gradient of this loss with respect to the input of this model. To do this, we will use the `gradients` function packaged with the `backend` module of Keras:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The call to `gradients` returns a list of tensors (of size 1 in this case)\n",
    "# hence we only keep the first element, which is a tensor.\n",
    "grads = K.gradients(loss, model.input)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A non-obvious trick to use for the gradient descent process to go smoothly is to normalize the gradient tensor, by dividing it by its L2 norm (the square root of the average of the square of the values in the tensor). This ensures that the magnitude of the updates done to the input image is always within a same range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We add 1e-5 before dividing so as to avoid accidentally dividing by 0.\n",
    "grads /= (K.sqrt(K.mean(K.square(grads))) + 1e-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need a way to compute the value of the loss tensor and the gradient tensor for a given input image. We can define a Keras backend function to do this: `iterate` is a function that takes a Numpy tensor (as a list of tensors of size 1) and returns a list of two Numpy tensors: the loss value and the gradient value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterate = K.function([model.input], [loss, grads])\n",
    "\n",
    "# Let's test it:\n",
    "import numpy as np\n",
    "loss_value, grads_value = iterate([np.zeros((1, 150, 150, 3))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "At this point, we can define a Python loop to do stochastic gradient descent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We start from a gray image with some noise\n",
    "input_img_data = np.random.random((1, 150, 150, 3)) * 20 + 128.\n",
    "\n",
    "# Run gradient ascent for 40 steps\n",
    "step = 1.  # this is the magnitude of each gradient update\n",
    "for i in range(40):\n",
    "    # Compute the loss value and gradient value\n",
    "    loss_value, grads_value = iterate([input_img_data])\n",
    "    # Here we adjust the input image in the direction that maximizes the loss\n",
    "    input_img_data += grads_value * step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting image tensor will be a floating point tensor of shape `(1, 150, 150, 3)`, with values that may not be integer within `[0, 255]`. Hence we would need to post-process this tensor to turn it into a displayable image. We do it with the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def deprocess_image(x):\n",
    "    # normalize tensor: center on 0., ensure std is 0.1\n",
    "    x -= x.mean()\n",
    "    x /= (x.std() + 1e-5)\n",
    "    x *= 0.1\n",
    "\n",
    "    # clip to [0, 1]\n",
    "    x += 0.5\n",
    "    x = np.clip(x, 0, 1)\n",
    "\n",
    "    # convert to RGB array\n",
    "    x *= 255\n",
    "    x = np.clip(x, 0, 255).astype('uint8')\n",
    "    return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all the pieces! Let's put them together into a Python function that takes as input a layer name and a filter index, and that returns a valid image tensor representing the pattern that maximizes the activation the specified filter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_pattern(layer_name, filter_index, size=150):\n",
    "    # Build a loss function that maximizes the activation of the nth filter of the layer considered.\n",
    "    layer_output = model.get_layer(layer_name).output\n",
    "    loss = K.mean(layer_output[:, :, :, filter_index])\n",
    "\n",
    "    # Compute the gradient of the input picture with this loss\n",
    "    grads = K.gradients(loss, model.input)[0]\n",
    "\n",
    "    # Normalization trick: we normalize the gradient\n",
    "    grads /= (K.sqrt(K.mean(K.square(grads))) + 1e-5)\n",
    "\n",
    "    # This function returns the loss and grads given the input picture\n",
    "    iterate = K.function([model.input], [loss, grads])\n",
    "    \n",
    "    # We start from a gray image with some noise\n",
    "    input_img_data = np.random.random((1, size, size, 3)) * 20 + 128.\n",
    "\n",
    "    # Run gradient ascent for 40 steps\n",
    "    step = 1.\n",
    "    for i in range(40):\n",
    "        loss_value, grads_value = iterate([input_img_data])\n",
    "        input_img_data += grads_value * step\n",
    "        \n",
    "    img = input_img_data[0]\n",
    "    return deprocess_image(img)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try this function for a few filters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 97340 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 104860 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 102960 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 106012 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for i in range(4):\n",
    "    plt.imshow(generate_pattern('block3_conv1', i))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that filter 0 in layer `block3_conv1` is responsive to a polka dot pattern.\n",
    "\n",
    "Now, let's visualize all filters in every layer. For simplicity, we will only look at the first 64 filters in each layer, and will only look at the first layer of each convolution block (block1_conv1, block2_conv1, block3_conv1, block4_conv1, \n",
    "block5_conv1). We will arrange the outputs on a 8x8 grid of 64x64 filter patterns, with some black margins between each filter pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block1_conv1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1336320 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block1_conv2\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1289480 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block2_conv1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1425860 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block2_conv2\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1359012 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block3_conv1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1387284 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block3_conv2\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1385872 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block4_conv1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1385668 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block4_conv2\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1413872 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block5_conv1\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1447340 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Layer name: block5_conv2\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 1482488 characters]\n",
      "text/plain": [
       "<Figure size 2304x2304 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "size = 64\n",
    "margin = 3\n",
    "no = 8\n",
    "\n",
    "for layer_name in ['block1_conv1', 'block1_conv2', 'block2_conv1', 'block2_conv2', 'block3_conv1', 'block3_conv2', 'block4_conv1', 'block4_conv2', 'block5_conv1', 'block5_conv2']: \n",
    "    \n",
    "    # This a empty (black) image where we will store our results.\n",
    "    results = np.zeros((no * size + 7 * margin, no * size + 7 * margin, 3), dtype=np.uint8)\n",
    "\n",
    "    for i in range(no):  # iterate over the rows of our results grid\n",
    "        for j in range(no):  # iterate over the columns of our results grid\n",
    "            # Generate the pattern for filter `i + (j * no)` in `layer_name`\n",
    "            filter_img = generate_pattern(layer_name, i + (j * no), size=size)\n",
    "            # Put the result in the square `(i, j)` of the results grid\n",
    "            horizontal_start = i * size + i * margin\n",
    "            horizontal_end = horizontal_start + size\n",
    "            vertical_start = j * size + j * margin\n",
    "            vertical_end = vertical_start + size\n",
    "            results[horizontal_start : horizontal_end, vertical_start : vertical_end, : ] = filter_img\n",
    "\n",
    "    # Display the results grid\n",
    "    print(\"Layer name: \" + layer_name)\n",
    "    plt.figure(figsize=(size / 2, size / 2))\n",
    "    plt.imshow(results)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These filter visualizations tell us a lot about how convnet layers see the world: each layer in a convnet simply learns a collection of filters such that their inputs can be expressed as a combination of the filters. This is similar to how the Fourier transform decomposes signals onto a bank of cosine functions. The filters in these convnet filter banks get increasingly complex and refined as we go higher-up in the model:\n",
    "\n",
    "* The filters from the first layer in the model (`block1_conv1`) encode simple directional edges and colors (or colored edges in some cases).\n",
    "* The filters from `block2_conv1` encode simple textures made from combinations of edges and colors.\n",
    "* The filters in higher-up layers start resembling textures found in natural images: feathers, eyes, leaves, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Visualizing heatmaps of class activation\n",
    "\n",
    "We will introduce a visualization technique that is useful for understanding which parts of a given image led a convnet to its final classification decision. This is helpful for \"debugging\" the decision process of a convnet, in particular in case of classification mistakes. It also allows you to locate specific objects in an image.\n",
    "\n",
    "This general category of techniques is called __Class Activation Map\" (CAM)__ visualization, and consists in producing heatmaps of \"class activation\" over input images. A \"class activation\" heatmap is a 2D grid of scores associated with an specific output class, computed for every location in any input image, indicating how important each location is with respect to the class considered. For instance, given a image fed into one of our \"cat vs. dog\" convnet, Class Activation Map visualization allows us to generate a heatmap for the class \"cat\", indicating how cat-like different parts of the image are, and likewise for the class \"dog\", indicating how dog-like differents parts of the image are.\n",
    "\n",
    "The specific implementation we will use is the one described in [Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization](https://arxiv.org/abs/1610.02391). It is very simple: it consists in taking the output feature map of a convolution layer given an input image, and weighing every channel in that feature map by the gradient of the class with respect to the channel. Intuitively, one way to understand this trick is that we are weighting a spatial map of \"how intensely the input image activates different channels\" by \"how important each channel is with regard to the class\", resulting in a spatial map of \"how intensely the input image activates the class\".\n",
    "\n",
    "We will demonstrate this technique using the pre-trained (original) VGG16 network again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.applications.vgg16 import VGG16\n",
    "\n",
    "K.clear_session()\n",
    "\n",
    "# Note that we are including the densely-connected classifier on top; all previous times, we were discarding it.\n",
    "model = VGG16(weights='imagenet')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's consider the following image of two African elephants, possible a mother and its cub, strolling in the savanna (under a Creative \n",
    "Commons license):\n",
    "\n",
    "![elephants](http://horzyk.pl/ml/images/creative_commons_elephant.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert this image into something the VGG16 model can read: the model was trained on images of size 224x244, preprocessed according \n",
    "to a few rules that are packaged in the utility function `keras.applications.vgg16.preprocess_input`. So we need to load the image, resize \n",
    "it to 224x224, convert it to a Numpy float32 tensor, and apply these pre-processing rules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing import image\n",
    "from keras.applications.vgg16 import preprocess_input, decode_predictions\n",
    "import numpy as np\n",
    "\n",
    "#base_dir_elephant = 'http://horzyk.pl/ml/data/creative_commons_elephant.jpg'\n",
    "base_dir_elephant = 'C:/Users/Adria/ml/data/'\n",
    "\n",
    "# The local path to our target image\n",
    "img_path = base_dir_elephant + 'creative_commons_elephant.jpg'\n",
    "\n",
    "# `img` is a PIL image of size 224x224\n",
    "img = image.load_img(img_path, target_size=(224, 224))\n",
    "\n",
    "# `x` is a float32 Numpy array of shape (224, 224, 3)\n",
    "x = image.img_to_array(img)\n",
    "\n",
    "# We add a dimension to transform our array into a \"batch\" of size (1, 224, 224, 3)\n",
    "x = np.expand_dims(x, axis=0)\n",
    "\n",
    "# Finally we preprocess the batch (this does channel-wise color normalization)\n",
    "x = preprocess_input(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted: [('n02504458', 'African_elephant', 0.909421), ('n01871265', 'tusker', 0.086182885), ('n02504013', 'Indian_elephant', 0.0043545826)]\n"
     ]
    }
   ],
   "source": [
    "preds = model.predict(x)\n",
    "print('Predicted:', decode_predictions(preds, top=3)[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The top-3 classes predicted for this image are:\n",
    "\n",
    "* African elephant (with 92.5% probability)\n",
    "* Tusker (with 7% probability)\n",
    "* Indian elephant (with 0.4% probability)\n",
    "\n",
    "Thus our network has recognized our image as containing an undetermined quantity of African elephants. The entry in the prediction vector \n",
    "that was maximally activated is the one corresponding to the \"African elephant\" class, at index 386:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "386"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(preds[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To visualize which parts of this image were the most \"African elephant\"-like, let's set up the Grad-CAM process:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is the \"african elephant\" entry in the prediction vector\n",
    "african_elephant_output = model.output[:, 386]\n",
    "\n",
    "# The is the output feature map of the `block5_conv3` layer,\n",
    "# the last convolutional layer in VGG16\n",
    "last_conv_layer = model.get_layer('block5_conv3')\n",
    "\n",
    "# This is the gradient of the \"african elephant\" class with regard to\n",
    "# the output feature map of `block5_conv3`\n",
    "grads = K.gradients(african_elephant_output, last_conv_layer.output)[0]\n",
    "\n",
    "# This is a vector of shape (512,), where each entry\n",
    "# is the mean intensity of the gradient over a specific feature map channel\n",
    "pooled_grads = K.mean(grads, axis=(0, 1, 2))\n",
    "\n",
    "# This function allows us to access the values of the quantities we just defined:\n",
    "# `pooled_grads` and the output feature map of `block5_conv3`,\n",
    "# given a sample image\n",
    "iterate = K.function([model.input], [pooled_grads, last_conv_layer.output[0]])\n",
    "\n",
    "# These are the values of these two quantities, as Numpy arrays,\n",
    "# given our sample image of two elephants\n",
    "pooled_grads_value, conv_layer_output_value = iterate([x])\n",
    "\n",
    "# We multiply each channel in the feature map array\n",
    "# by \"how important this channel is\" with regard to the elephant class\n",
    "for i in range(512):\n",
    "    conv_layer_output_value[:, :, i] *= pooled_grads_value[i]\n",
    "\n",
    "# The channel-wise mean of the resulting feature map\n",
    "# is our heatmap of class activation\n",
    "heatmap = np.mean(conv_layer_output_value, axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For visualization purpose, we will also normalize the heatmap between 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 5368 characters]\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "heatmap = np.maximum(heatmap, 0)\n",
    "heatmap /= np.max(heatmap)\n",
    "plt.matshow(heatmap)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will use OpenCV to generate an image that superimposes the original image with the heatmap we just obtained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 206172 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 30560 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 214848 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:/Users/Adria/ml/data/elephant_cam.jpg\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "\n",
    "# We use cv2 to load the original image\n",
    "img = cv2.imread(img_path)\n",
    "plt.imshow(img)\n",
    "plt.show()\n",
    "\n",
    "# We resize the heatmap to have the same size as the original image\n",
    "heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))\n",
    "\n",
    "# We convert the heatmap to RGB\n",
    "heatmap = np.uint8(255 * heatmap)\n",
    "\n",
    "# We apply the heatmap to the original image\n",
    "heatmap = cv2.applyColorMap(heatmap, cv2.COLORMAP_HOT)\n",
    "plt.imshow(heatmap)\n",
    "plt.show()\n",
    "\n",
    "# 0.4 here is a heatmap intensity factor\n",
    "superimposed_img = heatmap * 0.3 + img\n",
    "superimposed_img = superimposed_img.astype('uint8')\n",
    "\n",
    "plt.imshow(superimposed_img)\n",
    "plt.show()\n",
    "\n",
    "# Save the image to disk\n",
    "cv2.imwrite(base_dir_elephant + 'elephant_cam.jpg', superimposed_img)\n",
    "\n",
    "print(base_dir_elephant + 'elephant_cam.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![elephant cam](http://horzyk.pl/ml/images/elephant_cam.jpg)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This visualisation technique answers two important questions:\n",
    "\n",
    "* Why did the network think this image contained an African elephant?\n",
    "* Where is the African elephant located in the image?\n",
    "\n",
    "In particular, it is interesting to note that the ears of the elephant cub are strongly activated: this is probably how the network can recognize the difference between African and Indian elephants.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assignments\n",
    "\n",
    "* Try to visualize intermediate activations for a different image from this dataset (e.g. for a dog) and compare the visualized intermediate activation results.\n",
    "* Try tu use the `generate_pattern` function for different filter indices.\n",
    "* Use a different dataset from <a href=\"www.kaggle.com\">Kaggle.com</a> or another source and visualize:\n",
    "<ol>\n",
    "    <li>intermediate activations,</li>\n",
    "    <li>convnets filters,</li>\n",
    "    <li>heatmaps of class activation in an image.</li>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
