{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "<tr>    \n",
    "<td>\n",
    "<img src=\"http://home.agh.edu.pl/~horzyk/im/AdrianHorzyk140h.png\" alt=\"Adrian Horzyk, Professor\" title=\"Adrian Horzyk, Professor\" />        \n",
    "</td> \n",
    "<td style=\"text-align: center\">\n",
    "<h1>Introduction to Convolutional Neural Networks</h1>\n",
    "<h2><a href=\"http://home.agh.edu.pl/~horzyk/index-eng.php\">Adrian Horzyk</a></h2>\n",
    "<h3>Welcome to the interactive lecture where you can check everything by yourself and experiment!</h3>\n",
    "</td> \n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "\n",
    "became very popular because they can efficiently deal with many problems, especially with images that can be correctly recognized or classified independently of the movements of objects in the vision area, which was a big limitations of MLP networks (Dense layers only). \n",
    "\n",
    "Convolutional Neural Networks (CNN) can hierarchically extract important subpatterns, which are used to define high-order features that allow dense layers to classify input images with more ease and accuracy.\n",
    "\n",
    "Now, we will try to train such networks using the MNIST dataset to check whether we can achieve better performance and accuracy of the CNN-based models in comparison to the previous MLP (dense layers only) models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 2.1.0\n",
      "Keras version:2.3.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "'''Trains a simple ConvNet on the MNIST dataset. It gets over 99.6% validation accuracy after ~50 epochs\n",
    "(but there is still a margin for hyperparameter tuning). Training can take an hour or so!'''\n",
    "\n",
    "# Import libraries\n",
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "import math\n",
    "from math import ceil\n",
    "import tensorflow as tf\n",
    "import os   \n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt  # library for plotting math functions\n",
    "import pandas as pd\n",
    "import keras   # Import keras framework with various functions, models and structures\n",
    "from keras.datasets import mnist # gets MNIST dataset from repository\n",
    "from keras import layers\n",
    "from keras import models\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.callbacks import ReduceLROnPlateau\n",
    "from keras import optimizers\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import confusion_matrix, classification_report\n",
    "from keras.datasets import mnist\n",
    "from keras.utils import to_categorical\n",
    "from keras import regularizers\n",
    "\n",
    "# Set parameters for plots\n",
    "%matplotlib inline\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "print (\"TensorFlow version: \" + tf.__version__)\n",
    "print (\"Keras version:\" + keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create or choose directory for storing trained models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The path to the directory for storing and loading our models\n",
    "models_dir = 'C:/Users/Adria/ml/models/'\n",
    "if not os.path.exists(models_dir):\n",
    "    os.mkdir(models_dir)\n",
    "    \n",
    "# The path to the directory where the dataset are stored\n",
    "#original_dataset_dir = 'http://horzyk.pl/ml/data/kaggle/original/train'\n",
    "original_dataset_dir = 'C:/Users/Adria/ml/data/kaggle/original/train'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of classes:  10\n",
      "Distribution of the numbers training data of various classes:  (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8), array([5923, 6742, 5958, 6131, 5842, 5421, 5918, 6265, 5851, 5949],\n",
      "      dtype=int64))\n",
      "Distribution of the numbers validation data of various classes:  (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8), array([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009],\n",
      "      dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "# Split the data between train and validation sets. Here we are using only two sets for model training and validation.\n",
    "(x_train, y_train), (x_val, y_val) = mnist.load_data() # 60000 training and 10000 validating examples\n",
    "\n",
    "# Number of training and val data\n",
    "num_train_data = x_train.shape[0]\n",
    "num_val_data = x_val.shape[0]\n",
    "\n",
    "# Input image dimensions\n",
    "img_rows, img_cols = x_train.shape[1], x_train.shape[2]\n",
    "\n",
    "num_classes = np.size(np.unique(y_train))    # number of unique labels = number of classes of digits: 0, 1, 2, ..., 9\n",
    "print ('Number of classes: ', num_classes)\n",
    "\n",
    "## Distribution of the numbers training data of various classes\n",
    "print ('Distribution of the numbers training data of various classes: ', np.unique(y_train, return_counts=True))\n",
    "\n",
    "## Distribution of the numbers training data of various classes\n",
    "print ('Distribution of the numbers validation data of various classes: ', np.unique(y_val, return_counts=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few examples of the MNIST data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9060 characters]\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 132340 characters]\n",
      "text/plain": [
       "<Figure size 1440x720 with 200 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show a few sample digits from the training set\n",
    "plt.rcParams['figure.figsize'] = (2.5, 2.5) # set default size of plots\n",
    "col1 = 10\n",
    "row1 = 1\n",
    "fig = plt.figure(figsize=(col1, row1))\n",
    "for index in range(0, col1*row1):\n",
    "    fig.add_subplot(row1, col1, index + 1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_train[index]) # index of the sample picture\n",
    "    plt.title(\"Class \" + str(y_train[index]))\n",
    "plt.show()\n",
    "\n",
    "# Show a few sample digits from the training set\n",
    "plt.rcParams['figure.figsize'] = (1.0, 1.0) # set default size of plots\n",
    "col2 = 20\n",
    "row2 = 10\n",
    "fig = plt.figure(figsize=(col2, row2))\n",
    "for index in range(col1*row1, col1*row1 + col2*row2):\n",
    "    fig.add_subplot(row2, col2, index - col1*row1 + 1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_train[index]) # index of the sample picture\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reshape and transform the data to the expected types and ranges:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 validation samples\n"
     ]
    }
   ],
   "source": [
    "# According to the different formats reshape training and validation data\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_val = x_val.reshape(x_val.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_val = x_val.reshape(x_val.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "    \n",
    "# Transform training and validating data and show their shapes\n",
    "x_train = x_train.astype('float32')   # Copy this array and cast it to a specified type\n",
    "x_val = x_val.astype('float32')     # Copy this array and cast it to a specified type\n",
    "x_train /= 255   # Transfrom the training data from the range of 0 and 255 to the range of 0 and 1\n",
    "x_val /= 255    # Transfrom the validation data from the range of 0 and 255 to the range of 0 and 1\n",
    "\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_val.shape[0], 'validation samples')\n",
    "\n",
    "# Convert class vectors (integers) to binary class matrices using as specific \n",
    "y_train = keras.utils.to_categorical(y_train, num_classes) # y_train - a converted class vector into a matrix (integers from 0 to num_classes)\n",
    "y_val = keras.utils.to_categorical(y_val, num_classes) # y_val - a converted class vector into a matrix (integers from 0 to num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to ConvNets\n",
    "\n",
    "First, let's take a look at a very simple convnet example. We will use a convnet to classify MNIST digits, a task that you've already been that we already mentioned and achieved the validation accuracy ~98%). Even though our first convnet will be very basic, its accuracy will outperform the previous densely-connected model.\n",
    "\n",
    "To construct a basic convnet, we use a small stack of `Conv2D` and `MaxPooling2D` layers.\n",
    "\n",
    "This convnet takes as input tensors of shape `(image_height, image_width, image_channels)` (not including the batch dimension). We will configure our convnet to process inputs of size `(28, 28, 1)` via passing the argument `input_shape=(28, 28, 1)` to our first layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 3, 3, 64)          36928     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 576)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                36928     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 93,322\n",
      "Trainable params: 93,322\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# A simple CONVNET model for MNIST data\n",
    "model1 = models.Sequential()\n",
    "model1.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(img_rows, img_cols, 1)))\n",
    "model1.add(layers.MaxPooling2D((2, 2)))\n",
    "model1.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "model1.add(layers.MaxPooling2D((2, 2)))\n",
    "model1.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "# Flatten the output to input data to the Dense layer\n",
    "model1.add(layers.Flatten())\n",
    "# Dense layers - similar to the previous model, but using the smaller number of units\n",
    "model1.add(layers.Dense(64, activation='relu'))\n",
    "model1.add(layers.Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model1.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can notice that the output of every `Conv2D` and `MaxPooling2D` layer is a 3D tensor of shape `(height, width, channels)`. The width and height dimensions tend to shrink as we go deeper in the network. The number of channels is controlled by the first argument (hyperparameter) passed to the `Conv2D` layers (e.g., 32 or 64), and it is set by the developer (i.e., it is not automatically optimized by the model).\n",
    "\n",
    "The next step would be to feed our last output tensor (of shape `(3, 3, 64)`) into a densely-connected classifier network like those you are already familiar with - a stack of `Dense` layers, but the dense classifiers require a vector (1D tensor) on its input, so we must flatten our 3D outputs to 1D first, and then add a few `Dense` layers.\n",
    "\n",
    "As you can see, our `(3, 3, 64)` outputs were flattened into vectors of shape `(576,)`, before going through two `Dense` layers.\n",
    "\n",
    "Let's notice that our previous dense-based model had about 4 times more parameters (weights) that the above convnet model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's train our convnet on the MNIST digits and compare results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/50\n",
      "60000/60000 [==============================] - 19s 309us/step - loss: 0.3562 - acc: 0.8887 - val_loss: 0.1179 - val_acc: 0.9654\n",
      "Epoch 2/50\n",
      "60000/60000 [==============================] - 19s 323us/step - loss: 0.0773 - acc: 0.9756 - val_loss: 0.0576 - val_acc: 0.9825\n",
      "Epoch 3/50\n",
      "60000/60000 [==============================] - 21s 352us/step - loss: 0.0485 - acc: 0.9851 - val_loss: 0.0455 - val_acc: 0.9847\n",
      "Epoch 4/50\n",
      "60000/60000 [==============================] - 19s 318us/step - loss: 0.0354 - acc: 0.9887 - val_loss: 0.0325 - val_acc: 0.9889\n",
      "Epoch 5/50\n",
      "60000/60000 [==============================] - 24s 395us/step - loss: 0.0275 - acc: 0.9915 - val_loss: 0.0479 - val_acc: 0.9848\n",
      "Epoch 6/50\n",
      "60000/60000 [==============================] - 19s 314us/step - loss: 0.0227 - acc: 0.9926 - val_loss: 0.0246 - val_acc: 0.9910\n",
      "Epoch 7/50\n",
      "60000/60000 [==============================] - 18s 307us/step - loss: 0.0179 - acc: 0.9944 - val_loss: 0.0267 - val_acc: 0.9908\n",
      "Epoch 8/50\n",
      "60000/60000 [==============================] - 19s 313us/step - loss: 0.0159 - acc: 0.9950 - val_loss: 0.0237 - val_acc: 0.9924\n",
      "Epoch 9/50\n",
      "60000/60000 [==============================] - 17s 292us/step - loss: 0.0130 - acc: 0.9958 - val_loss: 0.0261 - val_acc: 0.9920\n",
      "Epoch 10/50\n",
      "60000/60000 [==============================] - 21s 348us/step - loss: 0.0108 - acc: 0.9966 - val_loss: 0.0331 - val_acc: 0.9891\n",
      "Epoch 11/50\n",
      "60000/60000 [==============================] - 19s 321us/step - loss: 0.0096 - acc: 0.9969 - val_loss: 0.0325 - val_acc: 0.9904\n",
      "Epoch 12/50\n",
      "60000/60000 [==============================] - 18s 300us/step - loss: 0.0078 - acc: 0.9976 - val_loss: 0.0314 - val_acc: 0.9909\n",
      "Epoch 13/50\n",
      "60000/60000 [==============================] - 19s 317us/step - loss: 0.0066 - acc: 0.9980 - val_loss: 0.0370 - val_acc: 0.9902\n",
      "Epoch 14/50\n",
      "60000/60000 [==============================] - 20s 337us/step - loss: 0.0056 - acc: 0.9983 - val_loss: 0.0302 - val_acc: 0.9917\n",
      "Epoch 15/50\n",
      "60000/60000 [==============================] - 18s 308us/step - loss: 0.0053 - acc: 0.9982 - val_loss: 0.0330 - val_acc: 0.9912\n",
      "Epoch 16/50\n",
      "60000/60000 [==============================] - 21s 343us/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0318 - val_acc: 0.9922\n",
      "Epoch 17/50\n",
      "60000/60000 [==============================] - 20s 341us/step - loss: 0.0039 - acc: 0.9985 - val_loss: 0.0393 - val_acc: 0.9911\n",
      "Epoch 18/50\n",
      "60000/60000 [==============================] - 23s 390us/step - loss: 0.0034 - acc: 0.9988 - val_loss: 0.0339 - val_acc: 0.9923\n",
      "Epoch 19/50\n",
      "60000/60000 [==============================] - 20s 339us/step - loss: 0.0042 - acc: 0.9986 - val_loss: 0.0376 - val_acc: 0.9923\n",
      "Epoch 20/50\n",
      "60000/60000 [==============================] - 24s 392us/step - loss: 0.0029 - acc: 0.9991 - val_loss: 0.0426 - val_acc: 0.9917\n",
      "Epoch 21/50\n",
      "60000/60000 [==============================] - 20s 330us/step - loss: 0.0028 - acc: 0.9991 - val_loss: 0.0456 - val_acc: 0.9910\n",
      "Epoch 22/50\n",
      "60000/60000 [==============================] - 20s 340us/step - loss: 0.0027 - acc: 0.9991 - val_loss: 0.0408 - val_acc: 0.9912\n",
      "Epoch 23/50\n",
      "60000/60000 [==============================] - 20s 341us/step - loss: 0.0026 - acc: 0.9992 - val_loss: 0.0507 - val_acc: 0.9917\n",
      "Epoch 24/50\n",
      "60000/60000 [==============================] - 19s 317us/step - loss: 0.0022 - acc: 0.9993 - val_loss: 0.0427 - val_acc: 0.9924\n",
      "Epoch 25/50\n",
      "60000/60000 [==============================] - 22s 358us/step - loss: 0.0018 - acc: 0.9994 - val_loss: 0.0571 - val_acc: 0.9892\n",
      "Epoch 26/50\n",
      "60000/60000 [==============================] - 20s 335us/step - loss: 0.0023 - acc: 0.9993 - val_loss: 0.0498 - val_acc: 0.9919\n",
      "Epoch 27/50\n",
      "60000/60000 [==============================] - 20s 332us/step - loss: 0.0015 - acc: 0.9994 - val_loss: 0.0514 - val_acc: 0.9908\n",
      "Epoch 28/50\n",
      "60000/60000 [==============================] - 21s 350us/step - loss: 0.0019 - acc: 0.9994 - val_loss: 0.0590 - val_acc: 0.9911\n",
      "Epoch 29/50\n",
      "60000/60000 [==============================] - 21s 349us/step - loss: 0.0020 - acc: 0.9994 - val_loss: 0.0516 - val_acc: 0.9914\n",
      "Epoch 30/50\n",
      "60000/60000 [==============================] - 19s 315us/step - loss: 0.0012 - acc: 0.9995 - val_loss: 0.0611 - val_acc: 0.9917\n",
      "Epoch 31/50\n",
      "60000/60000 [==============================] - 19s 321us/step - loss: 0.0017 - acc: 0.9995 - val_loss: 0.0603 - val_acc: 0.9922\n",
      "Epoch 32/50\n",
      "60000/60000 [==============================] - 19s 325us/step - loss: 0.0015 - acc: 0.9995 - val_loss: 0.0544 - val_acc: 0.9925\n",
      "Epoch 33/50\n",
      "60000/60000 [==============================] - 18s 305us/step - loss: 0.0012 - acc: 0.9996 - val_loss: 0.0608 - val_acc: 0.9917\n",
      "Epoch 34/50\n",
      "60000/60000 [==============================] - 19s 312us/step - loss: 0.0014 - acc: 0.9996 - val_loss: 0.0574 - val_acc: 0.9922\n",
      "Epoch 35/50\n",
      "60000/60000 [==============================] - 18s 298us/step - loss: 9.2525e-04 - acc: 0.9996 - val_loss: 0.0606 - val_acc: 0.9928\n",
      "Epoch 36/50\n",
      "60000/60000 [==============================] - 18s 307us/step - loss: 0.0017 - acc: 0.9995 - val_loss: 0.0542 - val_acc: 0.9925\n",
      "Epoch 37/50\n",
      "60000/60000 [==============================] - 18s 296us/step - loss: 0.0016 - acc: 0.9996 - val_loss: 0.0594 - val_acc: 0.9914\n",
      "Epoch 38/50\n",
      "60000/60000 [==============================] - 18s 304us/step - loss: 0.0010 - acc: 0.9997 - val_loss: 0.0634 - val_acc: 0.9923\n",
      "Epoch 39/50\n",
      "60000/60000 [==============================] - 18s 299us/step - loss: 0.0013 - acc: 0.9996 - val_loss: 0.0613 - val_acc: 0.9917\n",
      "Epoch 40/50\n",
      "60000/60000 [==============================] - 17s 289us/step - loss: 0.0011 - acc: 0.9996 - val_loss: 0.0692 - val_acc: 0.9911\n",
      "Epoch 41/50\n",
      "60000/60000 [==============================] - 18s 293us/step - loss: 0.0010 - acc: 0.9997 - val_loss: 0.0794 - val_acc: 0.9905\n",
      "Epoch 42/50\n",
      "60000/60000 [==============================] - 18s 298us/step - loss: 0.0012 - acc: 0.9996 - val_loss: 0.0612 - val_acc: 0.9925\n",
      "Epoch 43/50\n",
      "60000/60000 [==============================] - 18s 292us/step - loss: 0.0012 - acc: 0.9997 - val_loss: 0.0644 - val_acc: 0.9923\n",
      "Epoch 44/50\n",
      "60000/60000 [==============================] - 18s 300us/step - loss: 0.0012 - acc: 0.9997 - val_loss: 0.0775 - val_acc: 0.9915\n",
      "Epoch 45/50\n",
      "60000/60000 [==============================] - 18s 306us/step - loss: 9.7104e-04 - acc: 0.9997 - val_loss: 0.0681 - val_acc: 0.9918\n",
      "Epoch 46/50\n",
      "60000/60000 [==============================] - 18s 293us/step - loss: 7.4692e-04 - acc: 0.9998 - val_loss: 0.0639 - val_acc: 0.9919\n",
      "Epoch 47/50\n",
      "60000/60000 [==============================] - 18s 301us/step - loss: 0.0013 - acc: 0.9997 - val_loss: 0.0820 - val_acc: 0.9915\n",
      "Epoch 48/50\n",
      "60000/60000 [==============================] - 19s 313us/step - loss: 8.6663e-04 - acc: 0.9998 - val_loss: 0.0762 - val_acc: 0.9919\n",
      "Epoch 49/50\n",
      "60000/60000 [==============================] - 19s 318us/step - loss: 8.2048e-04 - acc: 0.9997 - val_loss: 0.0655 - val_acc: 0.9924\n",
      "Epoch 50/50\n",
      "60000/60000 [==============================] - 19s 313us/step - loss: 0.0011 - acc: 0.9997 - val_loss: 0.0657 - val_acc: 0.9924\n"
     ]
    }
   ],
   "source": [
    "# Set up the basic hyperparameters\n",
    "epochs = 50       # the number of training epochs\n",
    "batch_size = 256  # the size of mini-batches used for training the network in one step\n",
    "\n",
    "#early_stopping = EarlyStopping(monitor='val_loss', patience=2)\n",
    "#model.fit(x, y, validation_split=0.2, callbacks=[early_stopping])\n",
    "\n",
    "model1.compile(optimizer='rmsprop',\n",
    "                loss='categorical_crossentropy',\n",
    "                metrics=['acc'])\n",
    "\n",
    "# Save weights of the model before training to enable to start training it again\n",
    "model1.save_weights(models_dir + 'CNN_MNIST_1_weights_init.h5')\n",
    "\n",
    "history1 = model1.fit(x_train, y_train, epochs = epochs, batch_size = batch_size, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Save the trained model for future use or comparisons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "model1.save(models_dir + 'CNN_MNIST_1b.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 118us/step\n",
      "validation accuracy: 0.9923999905586243\n",
      "validation loss: 0.06572354659869217\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model and print out the final scores for the validation set\n",
    "val_loss1, val_acc1 = model1.evaluate(x_val, y_val)\n",
    "print('validation accuracy:', val_acc1)\n",
    "print('validation loss:', val_loss1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can notice that the training of convnet takes much longer time that for our dense-base network; however the achieved results on the validation set is much better. Now, we achieved ~99.3% instaed of ~98% after the same number of training epochs, using four times less parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best accuracy  0.965399980545044  for validation set we achieved in epoch:  0\n"
     ]
    }
   ],
   "source": [
    "argmax = np.argmin(history1.history['val_acc'])\n",
    "acc_max = history1.history['val_acc'][argmax]\n",
    "print ('The best accuracy ', acc_max, ' for validation set we achieved in epoch: ', argmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overfitting stated after epoch:  7 where the mininum loss was  0.02368794256723486\n"
     ]
    }
   ],
   "source": [
    "argmin = np.argmin(history1.history['val_loss'])\n",
    "loss_min = history1.history['val_loss'][argmin]\n",
    "print ('Overfitting stated after epoch: ', argmin, 'where the mininum loss was ', loss_min)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the number of training epochs to this value to stop training before overfitting and results on the validation set\n",
    "epochs = argmin       # the number of training epochs\n",
    "# and restart the training process again for this number of epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/7\n",
      "60000/60000 [==============================] - 18s 300us/step - loss: 0.3552 - acc: 0.8872\n",
      "Epoch 2/7\n",
      "60000/60000 [==============================] - 18s 295us/step - loss: 0.0755 - acc: 0.9763\n",
      "Epoch 3/7\n",
      "60000/60000 [==============================] - 17s 282us/step - loss: 0.0479 - acc: 0.9845\n",
      "Epoch 4/7\n",
      "60000/60000 [==============================] - 17s 282us/step - loss: 0.0354 - acc: 0.9887\n",
      "Epoch 5/7\n",
      "60000/60000 [==============================] - 19s 323us/step - loss: 0.0276 - acc: 0.9915\n",
      "Epoch 6/7\n",
      "60000/60000 [==============================] - 21s 357us/step - loss: 0.0231 - acc: 0.9927\n",
      "Epoch 7/7\n",
      "60000/60000 [==============================] - 22s 371us/step - loss: 0.0186 - acc: 0.9941\n"
     ]
    }
   ],
   "source": [
    "#model1.load_model(model_dir + 'CNN_MNIST_1a.h5')   # open the model which was saved before the start of its training\n",
    "model1.load_weights(models_dir + 'CNN_MNIST_1_weights_init.h5')\n",
    "\n",
    "model1.compile(optimizer='rmsprop',\n",
    "                loss='categorical_crossentropy',\n",
    "                metrics=['acc'])\n",
    "\n",
    "history1b = model1.fit(x_train, y_train, epochs = epochs, batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model1.save(models_dir + 'CNN_MNIST_1c.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 142us/step\n",
      "validation accuracy: 0.9904000163078308\n",
      "validation loss: 0.02783683568943161\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model and print out the final scores for the validation set\n",
    "val_loss1b, val_acc1b = model1.evaluate(x_val, y_val)\n",
    "print('validation accuracy:', val_acc1b)\n",
    "print('validation loss:', val_loss1b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next steps, we should try to modify our ConvNet to achieve event better results. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced ConvNet for MNIST Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we construct a deeper ConvNet with Dropout layers and use regularization.\n",
    "\n",
    "Next, we use training data augmentation and learning rate value control and adaptation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the network structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_4 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 13, 13, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 6, 6, 128)         73856     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 3, 3, 128)         0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 3, 3, 128)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 3, 3, 256)         295168    \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 3, 3, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 3, 3, 512)         1180160   \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 4608)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               589952    \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 2,175,754\n",
      "Trainable params: 2,175,754\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from keras import regularizers\n",
    "\n",
    "# Define the sequential Keras model composed of a few layers\n",
    "model2 = Sequential()   # establishes the type of the network model\n",
    "# Conv2D - creates a convolutional layer (https://keras.io/layers/convolutional/#conv2d) with \n",
    "# filters - specified number of convolutional filters\n",
    "# kernel_size - defines the frame (sliding window) size where the convolutional filter is implemented\n",
    "# activation - sets the activation function for this layers, here ReLU\n",
    "# input_shape - defines the shape of the input matrix (vector), here input_shape = (1, img_rows, img_cols)\n",
    "model2.add(Conv2D(filters=32, kernel_size=(3, 3),activation='relu', input_shape=input_shape))\n",
    "# model2.add(Conv2D(32, (3, 3), activation='relu')) - shoter way of the above code\n",
    "# MaxPooling2D pools the max value from the frame (sliding window) of 2 x 2 size\n",
    "model2.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model2.add(Dropout(0.20)) # Implements the drop out with the probability of 0.20\n",
    "model2.add(Conv2D(64, (3, 3), activation='relu',padding='same'))\n",
    "model2.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model2.add(Dropout(0.25))\n",
    "model2.add(Conv2D(128,(3, 3), activation='relu',padding='same'))\n",
    "model2.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model2.add(Dropout(0.30))\n",
    "model2.add(Conv2D(256,(3, 3), activation='relu',padding='same'))\n",
    "#model2.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model2.add(Dropout(0.40))\n",
    "model2.add(Conv2D(512,(3, 3), activation='relu',padding='same'))\n",
    "#model2.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model2.add(Dropout(0.50))\n",
    "# Finish the convolutional model and flatten the layer which does not affect the batch size.\n",
    "model2.add(Flatten())\n",
    "# Use a dense layer (MLP) consisting of 256 neurons with relu activation functions\n",
    "model2.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l1(0.001)))\n",
    "model2.add(Dropout(0.35))\n",
    "model2.add(Dense(128, activation='relu', kernel_regularizer=regularizers.l1(0.001)))\n",
    "model2.add(Dropout(0.25))\n",
    "model2.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "# Now print the full model, which will include the layers of the base model plus the dense layer we added\n",
    "print(model2.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile and train the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "234/234 [==============================] - 200s 855ms/step - loss: 2.6918 - acc: 0.5190 - val_loss: 0.6824 - val_acc: 0.9473\n",
      "Epoch 2/50\n",
      "234/234 [==============================] - 196s 837ms/step - loss: 0.7951 - acc: 0.8896 - val_loss: 0.4215 - val_acc: 0.9750\n",
      "Epoch 3/50\n",
      "234/234 [==============================] - 195s 832ms/step - loss: 0.5864 - acc: 0.9241 - val_loss: 0.3481 - val_acc: 0.9844\n",
      "Epoch 4/50\n",
      "234/234 [==============================] - 179s 763ms/step - loss: 0.4935 - acc: 0.9416 - val_loss: 0.3318 - val_acc: 0.9828\n",
      "Epoch 5/50\n",
      "234/234 [==============================] - 186s 793ms/step - loss: 0.4427 - acc: 0.9516 - val_loss: 0.2994 - val_acc: 0.9870\n",
      "Epoch 6/50\n",
      "234/234 [==============================] - 187s 800ms/step - loss: 0.4198 - acc: 0.9555 - val_loss: 0.2925 - val_acc: 0.9890\n",
      "Epoch 7/50\n",
      "234/234 [==============================] - 190s 810ms/step - loss: 0.3902 - acc: 0.9615 - val_loss: 0.2734 - val_acc: 0.9902\n",
      "Epoch 8/50\n",
      "234/234 [==============================] - 186s 796ms/step - loss: 0.3788 - acc: 0.9645 - val_loss: 0.2603 - val_acc: 0.9911\n",
      "Epoch 9/50\n",
      "234/234 [==============================] - 183s 783ms/step - loss: 0.3643 - acc: 0.9662 - val_loss: 0.2687 - val_acc: 0.9896\n",
      "Epoch 10/50\n",
      "234/234 [==============================] - 218s 933ms/step - loss: 0.3493 - acc: 0.9692 - val_loss: 0.2428 - val_acc: 0.9915\n",
      "Epoch 11/50\n",
      "234/234 [==============================] - 183s 784ms/step - loss: 0.3442 - acc: 0.9697 - val_loss: 0.2480 - val_acc: 0.9925\n",
      "Epoch 12/50\n",
      "234/234 [==============================] - 185s 790ms/step - loss: 0.3335 - acc: 0.9717 - val_loss: 0.2345 - val_acc: 0.9922\n",
      "Epoch 13/50\n",
      "234/234 [==============================] - 179s 763ms/step - loss: 0.3246 - acc: 0.9732 - val_loss: 0.2336 - val_acc: 0.9926\n",
      "Epoch 14/50\n",
      "234/234 [==============================] - 178s 759ms/step - loss: 0.3218 - acc: 0.9738 - val_loss: 0.2421 - val_acc: 0.9929\n",
      "Epoch 15/50\n",
      "234/234 [==============================] - 176s 752ms/step - loss: 0.3165 - acc: 0.9741 - val_loss: 0.2302 - val_acc: 0.9934\n",
      "Epoch 16/50\n",
      "234/234 [==============================] - 179s 766ms/step - loss: 0.3162 - acc: 0.9749 - val_loss: 0.2271 - val_acc: 0.9941\n",
      "Epoch 17/50\n",
      "234/234 [==============================] - 175s 748ms/step - loss: 0.3086 - acc: 0.9754 - val_loss: 0.2283 - val_acc: 0.9933\n",
      "Epoch 18/50\n",
      "234/234 [==============================] - 176s 752ms/step - loss: 0.3078 - acc: 0.9758 - val_loss: 0.2259 - val_acc: 0.9943\n",
      "Epoch 19/50\n",
      "234/234 [==============================] - 192s 822ms/step - loss: 0.2991 - acc: 0.9773 - val_loss: 0.2242 - val_acc: 0.9945\n",
      "Epoch 20/50\n",
      "234/234 [==============================] - 190s 811ms/step - loss: 0.2990 - acc: 0.9779 - val_loss: 0.2193 - val_acc: 0.9951\n",
      "Epoch 21/50\n",
      "234/234 [==============================] - 183s 784ms/step - loss: 0.2973 - acc: 0.9782 - val_loss: 0.2283 - val_acc: 0.9930\n",
      "Epoch 22/50\n",
      "234/234 [==============================] - 178s 761ms/step - loss: 0.2933 - acc: 0.9786 - val_loss: 0.2230 - val_acc: 0.9948\n",
      "Epoch 23/50\n",
      "234/234 [==============================] - 178s 762ms/step - loss: 0.2908 - acc: 0.9793 - val_loss: 0.2187 - val_acc: 0.9948\n",
      "Epoch 24/50\n",
      "234/234 [==============================] - 174s 745ms/step - loss: 0.2912 - acc: 0.9798 - val_loss: 0.2191 - val_acc: 0.9940\n",
      "Epoch 25/50\n",
      "234/234 [==============================] - 178s 759ms/step - loss: 0.2858 - acc: 0.9803 - val_loss: 0.2242 - val_acc: 0.9947\n",
      "\n",
      "Epoch 00025: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 26/50\n",
      "234/234 [==============================] - 180s 769ms/step - loss: 0.2918 - acc: 0.9791 - val_loss: 0.2251 - val_acc: 0.9943\n",
      "Epoch 27/50\n",
      "234/234 [==============================] - 163s 696ms/step - loss: 0.2802 - acc: 0.9805 - val_loss: 0.2149 - val_acc: 0.9952\n",
      "Epoch 28/50\n",
      "234/234 [==============================] - 165s 704ms/step - loss: 0.2783 - acc: 0.9814 - val_loss: 0.2362 - val_acc: 0.9959\n",
      "Epoch 29/50\n",
      "234/234 [==============================] - 162s 690ms/step - loss: 0.2851 - acc: 0.9808 - val_loss: 0.2144 - val_acc: 0.9956\n",
      "Epoch 30/50\n",
      "234/234 [==============================] - 165s 707ms/step - loss: 0.2830 - acc: 0.9808 - val_loss: 0.2334 - val_acc: 0.9953\n",
      "Epoch 31/50\n",
      "234/234 [==============================] - 182s 776ms/step - loss: 0.2821 - acc: 0.9814 - val_loss: 0.2223 - val_acc: 0.9956\n",
      "Epoch 32/50\n",
      "234/234 [==============================] - 182s 779ms/step - loss: 0.2777 - acc: 0.9816 - val_loss: 0.2118 - val_acc: 0.9954\n",
      "Epoch 33/50\n",
      "234/234 [==============================] - 183s 784ms/step - loss: 0.2749 - acc: 0.9821 - val_loss: 0.2161 - val_acc: 0.9943\n",
      "\n",
      "Epoch 00033: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 34/50\n",
      "234/234 [==============================] - 189s 808ms/step - loss: 0.2795 - acc: 0.9823 - val_loss: 0.2273 - val_acc: 0.9955\n",
      "Epoch 35/50\n",
      "234/234 [==============================] - 183s 780ms/step - loss: 0.2799 - acc: 0.9829 - val_loss: 0.2166 - val_acc: 0.9958\n",
      "Epoch 36/50\n",
      "234/234 [==============================] - 181s 772ms/step - loss: 0.2758 - acc: 0.9821 - val_loss: 0.2102 - val_acc: 0.9959\n",
      "Epoch 37/50\n",
      "234/234 [==============================] - 182s 778ms/step - loss: 0.2701 - acc: 0.9827 - val_loss: 0.2367 - val_acc: 0.9952\n",
      "Epoch 38/50\n",
      "234/234 [==============================] - 164s 699ms/step - loss: 0.2776 - acc: 0.9829 - val_loss: 0.2123 - val_acc: 0.9949\n",
      "\n",
      "Epoch 00038: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 39/50\n",
      "234/234 [==============================] - 167s 713ms/step - loss: 0.2626 - acc: 0.9837 - val_loss: 0.2087 - val_acc: 0.9954\n",
      "Epoch 40/50\n",
      "234/234 [==============================] - 163s 696ms/step - loss: 0.2754 - acc: 0.9828 - val_loss: 0.2176 - val_acc: 0.9954\n",
      "Epoch 41/50\n",
      "234/234 [==============================] - 165s 704ms/step - loss: 0.2744 - acc: 0.9828 - val_loss: 0.2361 - val_acc: 0.9945\n",
      "Epoch 42/50\n",
      "234/234 [==============================] - 166s 708ms/step - loss: 0.2743 - acc: 0.9835 - val_loss: 0.2232 - val_acc: 0.9958\n",
      "Epoch 43/50\n",
      "234/234 [==============================] - 165s 703ms/step - loss: 0.2585 - acc: 0.9841 - val_loss: 0.2007 - val_acc: 0.9952\n",
      "\n",
      "Epoch 00043: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 44/50\n",
      "234/234 [==============================] - 164s 701ms/step - loss: 0.2644 - acc: 0.9833 - val_loss: 0.2016 - val_acc: 0.9951\n",
      "Epoch 45/50\n",
      "234/234 [==============================] - 166s 709ms/step - loss: 0.2714 - acc: 0.9840 - val_loss: 0.2056 - val_acc: 0.9946\n",
      "Epoch 46/50\n",
      "234/234 [==============================] - 166s 707ms/step - loss: 0.2566 - acc: 0.9844 - val_loss: 0.2086 - val_acc: 0.9953\n",
      "Epoch 47/50\n",
      "234/234 [==============================] - 166s 708ms/step - loss: 0.2566 - acc: 0.9844 - val_loss: 0.2135 - val_acc: 0.9953\n",
      "Epoch 48/50\n",
      "234/234 [==============================] - 165s 705ms/step - loss: 0.2705 - acc: 0.9840 - val_loss: 0.2188 - val_acc: 0.9951\n",
      "\n",
      "Epoch 00048: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 49/50\n",
      "234/234 [==============================] - 161s 687ms/step - loss: 0.2687 - acc: 0.9844 - val_loss: 0.2161 - val_acc: 0.9954\n",
      "Epoch 50/50\n",
      "234/234 [==============================] - 162s 692ms/step - loss: 0.2556 - acc: 0.9851 - val_loss: 0.2112 - val_acc: 0.9958\n"
     ]
    }
   ],
   "source": [
    "# Define hyperparameters\n",
    "epochs = 50       # the number of training epochs\n",
    "batch_size = 256  # the size of mini-batches used for training the network in one step\n",
    "\n",
    "# Choose the optimizer that will be used: https://keras.io/api/optimizers/\n",
    "opt = optimizers.Adam(lr=0.001) #, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False, name=\"Adam\") # https://keras.io/api/optimizers/adam/\n",
    "#opt = optimizers.Adadelta(learning_rate=0.001, rho=0.95, epsilon=1e-07, name=\"Adadelta\", **kwargs)  # https://keras.io/api/optimizers/adadelta/\n",
    "#opt = optimizers.Adagrad(learning_rate=0.001, initial_accumulator_value=0.1, epsilon=1e-07, name=\"Adagrad\", **kwargs)  # https://keras.io/api/optimizers/adagrad/\n",
    "#opt = optimizers.Adamax(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, name=\"Adamax\", **kwargs)  # https://keras.io/api/optimizers/adamax/\n",
    "#opt = optimizers.Nadam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, name=\"Nadam\", **kwargs)  # https://keras.io/api/optimizers/Nadam/\n",
    "#opt = optimizers.Ftrl(learning_rate=0.001, learning_rate_power=-0.5, initial_accumulator_value=0.1, l1_regularization_strength=0.0, l2_regularization_strength=0.0, name=\"Ftrl\", l2_shrinkage_regularization_strength=0.0, beta=0.0, **kwargs)  # https://keras.io/api/optimizers/ftrl/\n",
    "#opt = optimizers.RMSprop(learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name=\"RMSprop\", **kwargs) # https://keras.io/api/optimizers/rmsprop/\n",
    "#opt = optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False, name=\"SGD\", **kwargs)  # https://keras.io/api/optimizers/sgd/\n",
    "\n",
    "# Compile the model using optimizer and the chosen metric: sparse_categorical_crossentropy or categorical_crossentropy etc.\n",
    "model2.compile(loss=keras.losses.categorical_crossentropy, # choose the metric: https://keras.io/api/metrics/\n",
    "              optimizer=opt,  # choose the optimizer\n",
    "              metrics=['acc']) # List of metrics to be evaluated by the model during training and validating: https://keras.io/models/model/#compile\n",
    "\n",
    "# Learning rate reduction during the training process: https://keras.io/callbacks/#reducelronplateau\n",
    "learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', # quantity to be monitored (val_loss)\n",
    "         factor=0.5, # factor by which the learning rate will be reduced. new_lr = lr * factor\n",
    "         patience=5, # number of epochs that produced the monitored quantity with no improvement after which training will be stopped\n",
    "         verbose=1,  # 0: quiet, 1: update messages.\n",
    "         min_lr=0.001) # lower bound on the learning rate\n",
    "\n",
    "# Augmentation of training data. It generates batches of tensor image data with real-time data augmentation. The data will be looped over in batches.\n",
    "datagen = ImageDataGenerator(\n",
    "        rotation_range=10, # rotate images in degrees up to the given degrees\n",
    "        zoom_range=0.2, # zoom images\n",
    "        width_shift_range=0.15,  # shift images horizontally\n",
    "        height_shift_range=0.15)  # shift images vertically\n",
    "# Computes the internal data stats related to the data-dependent transformations, based on an array of samples x_train\n",
    "datagen.fit(x_train) # Fits the data generator to the sample data x_train.\n",
    "\n",
    "# Simple train the model, validate, evaluate, and present scores\n",
    "'''history = model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,  # no of training epochs\n",
    "          verbose=1,  # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "          validation_data=(x_val, y_val),\n",
    "          validation_split=0.2,  # cross-validation split 1/5\n",
    "          shuffle=True) # method of how to shuffle training and validation data '''\n",
    "\n",
    "# Advanced train the model, validate, evaluate, and present scores: https://keras.io/models/model/#fit_generator\n",
    "history2 = model2.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),\n",
    "                            epochs=epochs,  # no of training epochs\n",
    "                            steps_per_epoch=x_train.shape[0]//batch_size, # no of mini-batches\n",
    "                            validation_data=(x_val, y_val),\n",
    "                            verbose=1,  # 0 = silent, 1 = progress bar, 2 = one line per epoch\n",
    "                            callbacks=[learning_rate_reduction])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Save the trained model for future use or comparisons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2.save(models_dir + 'CNN_MNIST_2.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate, score and plot the accuracy and the loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt  # library for plotting math functions: https://matplotlib.org/stable/index.html\n",
    "    \n",
    "def PlotTwoAccuracyComparison(acc1, val_acc1, acc2, val_acc2, lab1 = 'Model 1', lab2 = 'Model 2'):\n",
    "    #plt.clf()   # clear figure\n",
    "    plt.rcParams['image.interpolation'] = 'nearest'\n",
    "    plt.rcParams['image.cmap'] = 'gray'\n",
    "    plt.rcParams['figure.figsize'] = (15.0, 5.0) # set default size of plots\n",
    "    epochs = range(len(acc1))\n",
    "    plt.plot(epochs, acc1, 'bo', label='Training accuracy for ' + lab1)\n",
    "    plt.plot(epochs, val_acc1, 'b', label='Validation accuracy for ' + lab1)\n",
    "    plt.plot(epochs, acc2, 'ro', label='Training accuracy for ' + lab2)\n",
    "    plt.plot(epochs, val_acc2, 'r', label='Validation accuracy for ' + lab2)\n",
    "    plt.title('Comparison of Training and Validation Accuracies')\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.legend(['Train Acc 1', 'Val Acc 1', 'Train Acc 2', 'Val Acc 2'], loc='lower right')\n",
    "    plt.show()\n",
    "    \n",
    "def PlotTwoLossComparison(loss1, val_loss1, loss2, val_loss2, lab1 = 'Model 1', lab2 = 'Model 2'):\n",
    "    #plt.clf()   # clear figure\n",
    "    plt.rcParams['image.interpolation'] = 'nearest'\n",
    "    plt.rcParams['image.cmap'] = 'gray'\n",
    "    plt.rcParams['figure.figsize'] = (15.0, 5.0) # set default size of plots\n",
    "    epochs = range(len(loss1))\n",
    "    plt.plot(epochs, loss1, 'bo', label='Training loss for ' + lab1)\n",
    "    plt.plot(epochs, val_loss1, 'b', label='Validation loss for ' + lab1)\n",
    "    plt.plot(epochs, loss2, 'ro', label='Training loss for ' + lab2)\n",
    "    plt.plot(epochs, val_loss2, 'r', label='Validation loss for ' + lab2)\n",
    "    plt.title('Comparison of Training and Validation Losses')\n",
    "    plt.xlabel('Epochs')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.legend(['Train Loss 1', 'Val Loss 1', 'Train Loss 2', 'Val Loss 2'], loc='upper right')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25012 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compare the accuracies of three training processes\n",
    "acc1 = history1.history['acc']\n",
    "val_acc1 = history1.history['val_acc']\n",
    "acc2 = history2.history['acc']\n",
    "val_acc2 = history2.history['val_acc']\n",
    "\n",
    "PlotTwoAccuracyComparison(acc1, val_acc1, acc2, val_acc2, 'Model 1', 'Model 2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24964 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compare the losses of three training processes\n",
    "loss1 = history1.history['loss']\n",
    "val_loss1 = history1.history['val_loss']\n",
    "loss2 = history2.history['loss']\n",
    "val_loss2 = history2.history['val_loss']\n",
    "\n",
    "PlotTwoLossComparison(loss1, val_loss1, loss2, val_loss2, 'Model 1', 'Model 2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "LABELS= ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']\n",
    "\n",
    "# Define the confusion matrix for the results\n",
    "def show_confusion_matrix(validations, predictions, num_classes):\n",
    "    matrix = metrics.confusion_matrix(validations, predictions)\n",
    "    plt.figure(figsize=(num_classes, num_classes))\n",
    "    hm = sns.heatmap(matrix,\n",
    "                cmap='coolwarm',\n",
    "                linecolor='white',\n",
    "                linewidths=1,\n",
    "                xticklabels=LABELS,\n",
    "                yticklabels=LABELS,\n",
    "                annot=True,\n",
    "                fmt='d')\n",
    "    plt.yticks(rotation = 0)  # Don't rotate (vertically) the y-axis labels\n",
    "    #hm.invert_yaxis() # Invert the labels of the y-axis\n",
    "    hm.set_ylim(0, len(matrix))\n",
    "    plt.title('Confusion Matrix')\n",
    "    plt.ylabel('True Label')\n",
    "    plt.xlabel('Predicted Label')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.2111791367292404\n",
      "Validation accuracy: 0.9958000183105469\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 29776 characters]\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00       980\n",
      "           1       1.00      1.00      1.00      1135\n",
      "           2       1.00      1.00      1.00      1032\n",
      "           3       0.99      1.00      1.00      1010\n",
      "           4       0.99      1.00      1.00       982\n",
      "           5       1.00      0.99      0.99       892\n",
      "           6       1.00      0.99      1.00       958\n",
      "           7       0.99      1.00      0.99      1028\n",
      "           8       1.00      1.00      1.00       974\n",
      "           9       1.00      0.99      0.99      1009\n",
      "\n",
      "    accuracy                           1.00     10000\n",
      "   macro avg       1.00      1.00      1.00     10000\n",
      "weighted avg       1.00      1.00      1.00     10000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model and print out the final scores for the val set\n",
    "score = model2.evaluate(x_val, y_val, verbose=0)  # evaluate the model on the val set\n",
    "print('Validation loss:', score[0])      # print out the loss = score[0] (generalization error) \n",
    "print('Validation accuracy:', score[1])  # print out the generalization accuracy = score[1] of the model on val set\n",
    "\n",
    "# Use the trained model for predictions of the val data (in this case, we use validation data as val data)\n",
    "y_pred_val = model2.predict(x_val)\n",
    "\n",
    "# Take the class with the highest probability from the val predictions as a winning one\n",
    "max_y_pred_val = np.argmax(y_pred_val, axis=1)\n",
    "max_y_val = np.argmax(y_val, axis=1)\n",
    "\n",
    "# Show the confution matrix of the collected results\n",
    "show_confusion_matrix(max_y_val, max_y_pred_val, num_classes)\n",
    "\n",
    "# Print classification report\n",
    "print(classification_report(max_y_val, max_y_pred_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of misclassified examples:  42\n",
      "Misclassified examples:\n",
      "[ 359  625  674  716 1039 1232 1393 1681 1737 1878 1901 2035 2040 2130\n",
      " 2414 2462 2582 2597 3060 3422 3558 3726 3762 4163 4224 4284 4497 4507\n",
      " 4620 4699 5654 6571 6576 6597 6625 8316 8527 9638 9642 9692 9698 9729]\n",
      "Misclassified images (original class : predicted class):\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 42432 characters]\n",
      "text/plain": [
       "<Figure size 720x720 with 42 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Find out misclassified examples\n",
    "classcheck = max_y_val - max_y_pred_val  # 0 - when the class is the same, 1 - otherwise\n",
    "misclassified = np.where(classcheck != 0)[0]\n",
    "num_misclassified = len(misclassified)\n",
    "\n",
    "# Print misclassification report\n",
    "print('Number of misclassified examples: ', str(num_misclassified))\n",
    "print('Misclassified examples:')\n",
    "print(misclassified)\n",
    "\n",
    "# Show misclassified examples:\n",
    "print('Misclassified images (original class : predicted class):')\n",
    "plt.rcParams['figure.figsize'] = (2.5, 2.5) # set default size of plots\n",
    "col = 10\n",
    "row = 2 * math.ceil(num_misclassified / col)\n",
    "fig = plt.figure(figsize=(col, row))\n",
    "for index in range(0,num_misclassified):\n",
    "    fig.add_subplot(row, col, index + 1 + col*(index//col))\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_val[misclassified[index]].reshape(img_rows, img_cols)) # index of the val sample picture\n",
    "    plt.title(str(max_y_val[misclassified[index]]) + \":\" + str(max_y_pred_val[misclassified[index]]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The comparison of results has shown that our second model achieve higher accuracy than the simple model.\n",
    "\n",
    "You can still try to change hyperparameters of the model and achieve better results!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CIFAR-10 dataset training and validation example on ConvNet\n",
    "\n",
    "Let's look at another example of the convolutional neural network used to classify CIFAR-10 data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will use RGB images of 32 x 32 pixels and classify them into 10 classes.\n",
    "\n",
    "Training data consists of 50000 of images, and validating dataset of 10000 of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import cifar10 # gets CIFAR-10 dataset from repository\n",
    "\n",
    "# Define hyperparameters\n",
    "num_classes = 10 # number of classes of images\n",
    "\n",
    "# Input image dimensions\n",
    "img_rows, img_cols = 32, 32\n",
    "\n",
    "# Split the data between train and val sets\n",
    "(x_train, y_train), (x_val, y_val) = cifar10.load_data() # 50000 training and 10000 validating examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43464 characters]\n",
      "text/plain": [
       "<Figure size 720x72 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 776968 characters]\n",
      "text/plain": [
       "<Figure size 1440x720 with 200 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Show a few sample images from the training set\n",
    "plt.rcParams['figure.figsize'] = (2.5, 2.5) # set default size of plots\n",
    "col1 = 10\n",
    "row1 = 1\n",
    "fig = plt.figure(figsize=(col1, row1))\n",
    "for index in range(0, col1*row1):\n",
    "    fig.add_subplot(row1, col1, index + 1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_train[index]) # index of the sample picture\n",
    "    plt.title(\"Class \" + str(y_train[index]))\n",
    "plt.show()\n",
    "    \n",
    "# Show a few sample digits from the training set\n",
    "plt.rcParams['figure.figsize'] = (1.0, 1.0) # set default size of plots\n",
    "col2 = 20\n",
    "row2 = 10\n",
    "fig = plt.figure(figsize=(col2, row2))\n",
    "for index in range(col1*row1, col1*row1 + col2*row2):\n",
    "    fig.add_subplot(row2, col2, index - col1*row1 + 1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_train[index]) # index of the sample picture\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (50000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 validating samples\n"
     ]
    }
   ],
   "source": [
    "# According to the different formats reshape training and validating data\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 3, img_rows, img_cols)\n",
    "    x_val = x_val.reshape(x_val.shape[0], 3, img_rows, img_cols)\n",
    "    input_shape = (3, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 3)\n",
    "    x_val = x_val.reshape(x_val.shape[0], img_rows, img_cols, 3)\n",
    "    input_shape = (img_rows, img_cols, 3)\n",
    "\n",
    "# Transform training and validating data and show their shapes\n",
    "x_train = x_train.astype('float32')   # Copy this array and cast it to a specified type\n",
    "x_val = x_val.astype('float32')     # Copy this array and cast it to a specified type\n",
    "x_train /= 255 # Transfrom the training data from the range of 0 and 255 to the range of 0 and 1\n",
    "x_val /= 255  # Transfrom the validating data from the range of 0 and 255 to the range of 0 and 1\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_val.shape[0], 'validating samples')\n",
    "\n",
    "# Convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_val = keras.utils.to_categorical(y_val, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a model for this dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_4 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 13, 13, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 6, 6, 64)          0         \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 6, 6, 128)         73856     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_5 (MaxPooling2 (None, 3, 3, 128)         0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 3, 3, 128)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 3, 3, 256)         295168    \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 3, 3, 256)         0         \n",
      "_________________________________________________________________\n",
      "conv2d_8 (Conv2D)            (None, 3, 3, 512)         1180160   \n",
      "_________________________________________________________________\n",
      "dropout_5 (Dropout)          (None, 3, 3, 512)         0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 4608)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               589952    \n",
      "_________________________________________________________________\n",
      "dropout_6 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 128)               16512     \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 2,175,754\n",
      "Trainable params: 2,175,754\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Define the sequential Keras model composed of a few layers\n",
    "model3 = Sequential()   # establishes the type of the network model\n",
    "# Conv2D - creates a convolutional layer (https://keras.io/layers/convolutional/#conv2d) with \n",
    "# filters - specified number of convolutional filters\n",
    "# kernel_size - defines the frame (sliding window) size where the convolutional filter is implemented\n",
    "# activation - sets the activation function for this layers, here ReLU\n",
    "# input_shape - defines the shape of the input matrix (vector), here input_shape = (1, img_rows, img_cols)\n",
    "model3.add(Conv2D(64, kernel_size=(3, 3),activation='relu', input_shape=input_shape))\n",
    "model3.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "# MaxPooling2D pools the max value from the frame (sliding window) of 2 x 2 size\n",
    "model3.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model3.add(Dropout(0.25)) # Implements the drop out with the probability of 0.25\n",
    "model3.add(Conv2D(128, (3, 3), activation='relu',padding='same'))\n",
    "#model3.add(Conv2D(128, (3, 3), activation='relu',padding='same'))\n",
    "model3.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model3.add(Dropout(0.35))\n",
    "model3.add(Conv2D(256, (3, 3), activation='relu',padding='same'))\n",
    "#model3.add(Conv2D(256, (3, 3), activation='relu',padding='same'))\n",
    "model3.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model3.add(Dropout(0.35))\n",
    "# Finish the convolutional model and flatten the layer which does not affect the batch size.\n",
    "model3.add(Flatten())\n",
    "# Use a dense layer (MLP) consisting of 256 neurons with relu activation functions\n",
    "model3.add(Dense(512, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "model3.add(Dropout(0.5))\n",
    "model3.add(Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.001)))\n",
    "model3.add(Dropout(0.3))\n",
    "model3.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "# Now print the full model and notice the number of all trainable paramters\n",
    "print(model2.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "97/97 [==============================] - 150s 2s/step - loss: 3.2728 - acc: 0.1209 - val_loss: 2.9559 - val_acc: 0.2389\n",
      "Epoch 2/50\n",
      "97/97 [==============================] - 158s 2s/step - loss: 2.8232 - acc: 0.2086 - val_loss: 2.6113 - val_acc: 0.2743\n",
      "Epoch 3/50\n",
      "97/97 [==============================] - 164s 2s/step - loss: 2.5122 - acc: 0.2603 - val_loss: 2.2373 - val_acc: 0.3421\n",
      "Epoch 4/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 2.2422 - acc: 0.3118 - val_loss: 1.9984 - val_acc: 0.3964\n",
      "Epoch 5/50\n",
      "97/97 [==============================] - 155s 2s/step - loss: 2.0604 - acc: 0.3499 - val_loss: 2.0067 - val_acc: 0.3689\n",
      "Epoch 6/50\n",
      "97/97 [==============================] - 167s 2s/step - loss: 1.9289 - acc: 0.3738 - val_loss: 1.7516 - val_acc: 0.4354\n",
      "Epoch 7/50\n",
      "97/97 [==============================] - 160s 2s/step - loss: 1.8226 - acc: 0.4009 - val_loss: 1.6646 - val_acc: 0.4544\n",
      "Epoch 8/50\n",
      "97/97 [==============================] - 164s 2s/step - loss: 1.7438 - acc: 0.4208 - val_loss: 1.5778 - val_acc: 0.4817\n",
      "Epoch 9/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 1.6590 - acc: 0.4421 - val_loss: 1.4451 - val_acc: 0.5172\n",
      "Epoch 10/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 1.6114 - acc: 0.4590 - val_loss: 1.4024 - val_acc: 0.5337\n",
      "Epoch 11/50\n",
      "97/97 [==============================] - 165s 2s/step - loss: 1.5523 - acc: 0.4803 - val_loss: 1.4943 - val_acc: 0.4907\n",
      "Epoch 12/50\n",
      "97/97 [==============================] - 159s 2s/step - loss: 1.5076 - acc: 0.4992 - val_loss: 1.4572 - val_acc: 0.5201\n",
      "Epoch 13/50\n",
      "97/97 [==============================] - 155s 2s/step - loss: 1.4519 - acc: 0.5182 - val_loss: 1.3147 - val_acc: 0.5673\n",
      "Epoch 14/50\n",
      "97/97 [==============================] - 165s 2s/step - loss: 1.4302 - acc: 0.5268 - val_loss: 1.2185 - val_acc: 0.6061\n",
      "Epoch 15/50\n",
      "97/97 [==============================] - 176s 2s/step - loss: 1.3968 - acc: 0.5415 - val_loss: 1.2040 - val_acc: 0.6172\n",
      "Epoch 16/50\n",
      "97/97 [==============================] - 181s 2s/step - loss: 1.3650 - acc: 0.5507 - val_loss: 1.1472 - val_acc: 0.6397\n",
      "Epoch 17/50\n",
      "97/97 [==============================] - 176s 2s/step - loss: 1.3142 - acc: 0.5705 - val_loss: 1.1039 - val_acc: 0.6481\n",
      "Epoch 18/50\n",
      "97/97 [==============================] - 180s 2s/step - loss: 1.3066 - acc: 0.5789 - val_loss: 1.0319 - val_acc: 0.6767\n",
      "Epoch 19/50\n",
      "97/97 [==============================] - 177s 2s/step - loss: 1.2558 - acc: 0.5950 - val_loss: 1.0088 - val_acc: 0.6885\n",
      "Epoch 20/50\n",
      "97/97 [==============================] - 181s 2s/step - loss: 1.2449 - acc: 0.6026 - val_loss: 1.0300 - val_acc: 0.6880\n",
      "Epoch 21/50\n",
      "97/97 [==============================] - 182s 2s/step - loss: 1.2153 - acc: 0.6162 - val_loss: 1.0708 - val_acc: 0.6669\n",
      "Epoch 22/50\n",
      "97/97 [==============================] - 181s 2s/step - loss: 1.2145 - acc: 0.6144 - val_loss: 1.0004 - val_acc: 0.6904\n",
      "Epoch 23/50\n",
      "97/97 [==============================] - 179s 2s/step - loss: 1.1676 - acc: 0.6343 - val_loss: 1.0353 - val_acc: 0.6885\n",
      "Epoch 24/50\n",
      "97/97 [==============================] - 179s 2s/step - loss: 1.1698 - acc: 0.6378 - val_loss: 0.9957 - val_acc: 0.6986\n",
      "Epoch 25/50\n",
      "97/97 [==============================] - 181s 2s/step - loss: 1.1369 - acc: 0.6469 - val_loss: 0.9228 - val_acc: 0.7271\n",
      "Epoch 26/50\n",
      "97/97 [==============================] - 171s 2s/step - loss: 1.1275 - acc: 0.6516 - val_loss: 0.9664 - val_acc: 0.7103\n",
      "Epoch 27/50\n",
      "97/97 [==============================] - 163s 2s/step - loss: 1.1126 - acc: 0.6601 - val_loss: 0.8788 - val_acc: 0.7414\n",
      "Epoch 28/50\n",
      "97/97 [==============================] - 168s 2s/step - loss: 1.0989 - acc: 0.6640 - val_loss: 0.9005 - val_acc: 0.7352\n",
      "Epoch 29/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 1.0732 - acc: 0.6727 - val_loss: 1.0738 - val_acc: 0.6768\n",
      "Epoch 30/50\n",
      "97/97 [==============================] - 164s 2s/step - loss: 1.0699 - acc: 0.6750 - val_loss: 0.9519 - val_acc: 0.7273\n",
      "Epoch 31/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 1.0631 - acc: 0.6799 - val_loss: 1.0001 - val_acc: 0.6985\n",
      "Epoch 32/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 1.0411 - acc: 0.6847 - val_loss: 0.8557 - val_acc: 0.7405\n",
      "\n",
      "Epoch 00032: ReduceLROnPlateau reducing learning rate to 0.5.\n",
      "Epoch 33/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 0.9324 - acc: 0.7215 - val_loss: 0.7982 - val_acc: 0.7674\n",
      "Epoch 34/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 0.9165 - acc: 0.7254 - val_loss: 0.7769 - val_acc: 0.7724\n",
      "Epoch 35/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 0.9069 - acc: 0.7263 - val_loss: 0.7530 - val_acc: 0.7816\n",
      "Epoch 36/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 0.9049 - acc: 0.7279 - val_loss: 0.7881 - val_acc: 0.7706\n",
      "Epoch 37/50\n",
      "97/97 [==============================] - 160s 2s/step - loss: 0.8913 - acc: 0.7340 - val_loss: 0.7860 - val_acc: 0.7656\n",
      "Epoch 38/50\n",
      "97/97 [==============================] - 163s 2s/step - loss: 0.8890 - acc: 0.7302 - val_loss: 0.7666 - val_acc: 0.7762\n",
      "Epoch 39/50\n",
      "97/97 [==============================] - 158s 2s/step - loss: 0.8777 - acc: 0.7348 - val_loss: 0.7938 - val_acc: 0.7707\n",
      "Epoch 40/50\n",
      "97/97 [==============================] - 158s 2s/step - loss: 0.8683 - acc: 0.7368 - val_loss: 0.7113 - val_acc: 0.7910\n",
      "Epoch 41/50\n",
      "97/97 [==============================] - 159s 2s/step - loss: 0.8618 - acc: 0.7394 - val_loss: 0.7288 - val_acc: 0.7873\n",
      "Epoch 42/50\n",
      "97/97 [==============================] - 158s 2s/step - loss: 0.8624 - acc: 0.7373 - val_loss: 0.6872 - val_acc: 0.8040\n",
      "Epoch 43/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 0.8592 - acc: 0.7407 - val_loss: 0.7091 - val_acc: 0.7958\n",
      "Epoch 44/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 0.8412 - acc: 0.7452 - val_loss: 0.6973 - val_acc: 0.7999\n",
      "Epoch 45/50\n",
      "97/97 [==============================] - 161s 2s/step - loss: 0.8452 - acc: 0.7473 - val_loss: 0.7175 - val_acc: 0.7915\n",
      "Epoch 46/50\n",
      "97/97 [==============================] - 160s 2s/step - loss: 0.8400 - acc: 0.7484 - val_loss: 0.7323 - val_acc: 0.7856\n",
      "Epoch 47/50\n",
      "97/97 [==============================] - 174s 2s/step - loss: 0.8401 - acc: 0.7464 - val_loss: 0.7051 - val_acc: 0.7978\n",
      "\n",
      "Epoch 00047: ReduceLROnPlateau reducing learning rate to 0.25.\n",
      "Epoch 48/50\n",
      "97/97 [==============================] - 162s 2s/step - loss: 0.7937 - acc: 0.7630 - val_loss: 0.6772 - val_acc: 0.8042\n",
      "Epoch 49/50\n",
      "97/97 [==============================] - 166s 2s/step - loss: 0.7880 - acc: 0.7645 - val_loss: 0.6561 - val_acc: 0.8097\n",
      "Epoch 50/50\n",
      "97/97 [==============================] - 170s 2s/step - loss: 0.7703 - acc: 0.7691 - val_loss: 0.6623 - val_acc: 0.8083\n"
     ]
    }
   ],
   "source": [
    "# Define hyperparameters\n",
    "batch_size = 512 # size of mini-baches\n",
    "epochs = 50      # how many times all traing examples will be used to train the model\n",
    "\n",
    "# Compile the model using optimizer\n",
    "opt = optimizers.Adam(lr=0.001) #, beta_1=0.9, beta_2=0.999, epsilon=1e-07, amsgrad=False, name=\"Adam\") # https://keras.io/api/optimizers/adam/\n",
    "#opt = optimizers.Adadelta(learning_rate=0.001, rho=0.95, epsilon=1e-07, name=\"Adadelta\", **kwargs)  # https://keras.io/api/optimizers/adadelta/\n",
    "#opt = optimizers.Adagrad(learning_rate=0.001, initial_accumulator_value=0.1, epsilon=1e-07, name=\"Adagrad\", **kwargs)  # https://keras.io/api/optimizers/adagrad/\n",
    "#opt = optimizers.Adamax(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, name=\"Adamax\", **kwargs)  # https://keras.io/api/optimizers/adamax/\n",
    "#opt = optimizers.Nadam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-07, name=\"Nadam\", **kwargs)  # https://keras.io/api/optimizers/Nadam/\n",
    "#opt = optimizers.Ftrl(learning_rate=0.001, learning_rate_power=-0.5, initial_accumulator_value=0.1, l1_regularization_strength=0.0, l2_regularization_strength=0.0, name=\"Ftrl\", l2_shrinkage_regularization_strength=0.0, beta=0.0, **kwargs)  # https://keras.io/api/optimizers/ftrl/\n",
    "#opt = optimizers.RMSprop(learning_rate=0.001, rho=0.9, momentum=0.0, epsilon=1e-07, centered=False, name=\"RMSprop\", **kwargs) # https://keras.io/api/optimizers/rmsprop/\n",
    "#opt = optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False, name=\"SGD\", **kwargs)  # https://keras.io/api/optimizers/sgd/\n",
    "model3.compile(loss=keras.losses.categorical_crossentropy,\n",
    "               optimizer=opt,\n",
    "               metrics=['acc']) # List of metrics to be evaluated by the model during training and validating: https://keras.io/models/model/#compile\n",
    "\n",
    "# Learning rate reduction durint the training process: https://keras.io/callbacks/#reducelronplateau\n",
    "learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', # quantity to be monitored (val_loss)\n",
    "                                            factor=0.5, # factor by which the learning rate will be reduced. new_lr = lr * factor\n",
    "                                            patience=5, # number of epochs that produced the monitored quantity with no improvement after which training will be stopped\n",
    "                                            verbose=1,  # 0: quiet, 1: update messages.\n",
    "                                            min_lr=0.001) # lower bound on the learning rate\n",
    "\n",
    "# Augmentation of training data. It generates batches of tensor image data with real-time data augmentation. The data will be looped over in batches.\n",
    "datagen = ImageDataGenerator(\n",
    "          rotation_range=20,       # rotate images in degrees up to the given degrees\n",
    "          width_shift_range=0.1,   # shift images horizontally\n",
    "          height_shift_range=0.1,  # shift images vertically\n",
    "          horizontal_flip=True)    # flip images (left<->right)\n",
    "# Computes the internal data stats related to the data-dependent transformations, based on an array of samples x_train\n",
    "datagen.fit(x_train)\n",
    "\n",
    "# Train the model, validate, evaluate, and present scores\n",
    "history=model3.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),\n",
    "                            epochs=epochs,\n",
    "                            steps_per_epoch=x_train.shape[0]//batch_size,  # no of mini-batches \n",
    "                            validation_data=(x_val, y_val),\n",
    "                            verbose=1,\n",
    "                            callbacks=[learning_rate_reduction])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save weights of the model before training to enable to start training it again\n",
    "model3.save(models_dir + 'CNN_CIFAR-10_1.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loss: 0.5659338885784149\n",
      "Train accuracy: 0.8405399918556213\n",
      "Bias: 0.8405399918556213\n",
      "Validation loss: 0.6622600614547729\n",
      "Validation accuracy: 0.8083000183105469\n",
      "Variance: -0.03223997354507446\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 33056 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 29564 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "score_train = model3.evaluate(x_train, y_train, verbose=0)\n",
    "print('Train loss:', score_train[0])\n",
    "print('Train accuracy:', score_train[1])\n",
    "print('Bias:', score_train[1])   # Assuming that the Bayes optimal error is equal to 0.\n",
    "\n",
    "score_val = model3.evaluate(x_val, y_val, verbose=0)\n",
    "print('Validation loss:', score_val[0])\n",
    "print('Validation accuracy:', score_val[1])\n",
    "print('Variance:', score_val[1] - score_train[1])\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (15.0, 5.0) # set default size of plots\n",
    "\n",
    "# Plot training & validation accuracy values: https://keras.io/visualization/#training-history-visualization\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('Model accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(['Train', 'Val'], loc='upper left')  # OR plt.legend(['Train', 'Validation'], loc='upper left')\n",
    "plt.show()\n",
    "\n",
    "# Plot training & validation loss values: https://keras.io/visualization/#training-history-visualizatio\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('Model loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend(['Train', 'Val'], loc='lower left')  # OR plt.legend(['Train', 'Validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can notice, the variance is negative (less than 0). Should we be happy about this? Where it comes from? Mini-batch mode and regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at the validation time, so the model does not change as during the testing time. That is why the training error is always bigger, which can appear weird in view of the classic machine learning models. On the other hand, such calculated variance is not true. We had to calculate it without the turned-on regularization factors and with one batch to be comparable to the error calculated for testing data.\n",
    "\n",
    "The training and validation errors still decrease, so the training should be continued to achieve probably better results, decreasing bias, which is after 50 epochs not satisfactory!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 44484 characters]\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.87      0.84      1000\n",
      "           1       0.92      0.91      0.91      1000\n",
      "           2       0.86      0.64      0.73      1000\n",
      "           3       0.74      0.56      0.64      1000\n",
      "           4       0.75      0.80      0.78      1000\n",
      "           5       0.79      0.68      0.73      1000\n",
      "           6       0.73      0.94      0.82      1000\n",
      "           7       0.79      0.89      0.83      1000\n",
      "           8       0.91      0.88      0.89      1000\n",
      "           9       0.81      0.93      0.87      1000\n",
      "\n",
      "    accuracy                           0.81     10000\n",
      "   macro avg       0.81      0.81      0.80     10000\n",
      "weighted avg       0.81      0.81      0.80     10000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Use the trained model for predictions of the val data (in this case, we use validation data as val data)\n",
    "y_pred_val = model3.predict(x_val)\n",
    "\n",
    "# Take the class with the highest probability from the val predictions as a winning one\n",
    "max_y_pred_val = np.argmax(y_pred_val, axis=1)\n",
    "max_y_val = np.argmax(y_val, axis=1)\n",
    "\n",
    "# Show the confution matrix of the collected results\n",
    "show_confusion_matrix(max_y_val, max_y_pred_val, num_classes)\n",
    "\n",
    "# Print classification report\n",
    "print(classification_report(max_y_val, max_y_pred_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of misclassified examples:  1917\n",
      "Misclassified examples:\n",
      "[  12   24   25 ... 9989 9993 9996]\n",
      "Misclassified images (original class : predicted class):\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 8163472 characters]\n",
      "text/plain": [
       "<Figure size 720x27648 with 1917 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Find out misclassified examples\n",
    "classcheck = max_y_val - max_y_pred_val  # 0 - when the class is the same, 1 - otherwise\n",
    "misclassified = np.where(classcheck != 0)[0]\n",
    "num_misclassified = len(misclassified)\n",
    "\n",
    "# Print misclassification report\n",
    "print('Number of misclassified examples: ', str(num_misclassified))\n",
    "print('Misclassified examples:')\n",
    "print(misclassified)\n",
    "\n",
    "# Show misclassified examples:\n",
    "print('Misclassified images (original class : predicted class):')\n",
    "plt.rcParams['figure.figsize'] = (2.5, 2.5) # set default size of plots\n",
    "col = 10\n",
    "row = 2 * math.ceil(num_misclassified / col)\n",
    "fig = plt.figure(figsize=(col, row))\n",
    "for index in range(0,num_misclassified):\n",
    "    fig.add_subplot(row, col, index + 1 + col*(index//col))\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_val[misclassified[index]]) # index of the validation sample picture\n",
    "    plt.title(str(max_y_val[misclassified[index]]) + \":\" + str(max_y_pred_val[misclassified[index]]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we could see, training this dataset is not so easy as training the MNIST dataset.\n",
    "\n",
    "The bias of the model is still high, so the model under fits. However, the variance of the model seems not to be bad; however, we didn't calculate the real variance here, so it can be misleading! So, what do we need to do now with this model to reduce the high bias? Let's learn the same model longer (for another hundred of epochs) and change the mini-batch size from 512 to 256 to give the model the ability not to stack in local minima or saddle points so quickly.\n",
    "\n",
    "So now we apply __a babysitting model__ (__Panda strategy__) to improve the model performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "195/195 [==============================] - 173s 889ms/step - loss: 0.9736 - acc: 0.7079 - val_loss: 0.8086 - val_acc: 0.7718\n",
      "Epoch 2/100\n",
      "195/195 [==============================] - 173s 886ms/step - loss: 0.9700 - acc: 0.7178 - val_loss: 0.8519 - val_acc: 0.7654\n",
      "Epoch 3/100\n",
      "195/195 [==============================] - 169s 865ms/step - loss: 0.9819 - acc: 0.7179 - val_loss: 0.8258 - val_acc: 0.7761\n",
      "Epoch 4/100\n",
      "195/195 [==============================] - 170s 869ms/step - loss: 0.9722 - acc: 0.7245 - val_loss: 0.8943 - val_acc: 0.7561\n",
      "Epoch 5/100\n",
      "195/195 [==============================] - 172s 880ms/step - loss: 0.9747 - acc: 0.7243 - val_loss: 0.8011 - val_acc: 0.7848\n",
      "Epoch 6/100\n",
      "195/195 [==============================] - 166s 852ms/step - loss: 0.9656 - acc: 0.7287 - val_loss: 0.8172 - val_acc: 0.7821\n",
      "Epoch 7/100\n",
      "195/195 [==============================] - 167s 854ms/step - loss: 0.9497 - acc: 0.7358 - val_loss: 0.7600 - val_acc: 0.8059\n",
      "Epoch 8/100\n",
      "195/195 [==============================] - 166s 853ms/step - loss: 0.9354 - acc: 0.7398 - val_loss: 0.8046 - val_acc: 0.7881\n",
      "Epoch 9/100\n",
      "195/195 [==============================] - 170s 870ms/step - loss: 0.9440 - acc: 0.7359 - val_loss: 0.7823 - val_acc: 0.7917\n",
      "Epoch 10/100\n",
      "195/195 [==============================] - 165s 847ms/step - loss: 0.9326 - acc: 0.7404 - val_loss: 0.7751 - val_acc: 0.7983\n",
      "Epoch 11/100\n",
      "195/195 [==============================] - 166s 851ms/step - loss: 0.9292 - acc: 0.7440 - val_loss: 0.7771 - val_acc: 0.7938\n",
      "Epoch 12/100\n",
      "195/195 [==============================] - 165s 845ms/step - loss: 0.9146 - acc: 0.7444 - val_loss: 0.7812 - val_acc: 0.7962\n",
      "\n",
      "Epoch 00012: ReduceLROnPlateau reducing learning rate to 0.5.\n",
      "Epoch 13/100\n",
      "195/195 [==============================] - 170s 871ms/step - loss: 0.8295 - acc: 0.7733 - val_loss: 0.7202 - val_acc: 0.8078\n",
      "Epoch 14/100\n",
      "195/195 [==============================] - 166s 850ms/step - loss: 0.8116 - acc: 0.7731 - val_loss: 0.6650 - val_acc: 0.8229\n",
      "Epoch 15/100\n",
      "195/195 [==============================] - 167s 859ms/step - loss: 0.7926 - acc: 0.7802 - val_loss: 0.6546 - val_acc: 0.8254\n",
      "Epoch 16/100\n",
      "195/195 [==============================] - 166s 852ms/step - loss: 0.7778 - acc: 0.7821 - val_loss: 0.6565 - val_acc: 0.8228\n",
      "Epoch 17/100\n",
      "195/195 [==============================] - 168s 863ms/step - loss: 0.7761 - acc: 0.7800 - val_loss: 0.6659 - val_acc: 0.8205\n",
      "Epoch 18/100\n",
      "195/195 [==============================] - 166s 850ms/step - loss: 0.7678 - acc: 0.7823 - val_loss: 0.6532 - val_acc: 0.8206\n",
      "Epoch 19/100\n",
      "195/195 [==============================] - 169s 867ms/step - loss: 0.7670 - acc: 0.7802 - val_loss: 0.6269 - val_acc: 0.8290\n",
      "Epoch 20/100\n",
      "195/195 [==============================] - 169s 864ms/step - loss: 0.7585 - acc: 0.7848 - val_loss: 0.6438 - val_acc: 0.8224\n",
      "Epoch 21/100\n",
      "195/195 [==============================] - 177s 909ms/step - loss: 0.7574 - acc: 0.7814 - val_loss: 0.6066 - val_acc: 0.8347\n",
      "Epoch 22/100\n",
      "195/195 [==============================] - 219s 1s/step - loss: 0.7430 - acc: 0.7868 - val_loss: 0.6558 - val_acc: 0.8192\n",
      "Epoch 23/100\n",
      "195/195 [==============================] - 231s 1s/step - loss: 0.7494 - acc: 0.7857 - val_loss: 0.6262 - val_acc: 0.8261\n",
      "Epoch 24/100\n",
      "195/195 [==============================] - 215s 1s/step - loss: 0.7394 - acc: 0.7878 - val_loss: 0.6304 - val_acc: 0.8233\n",
      "Epoch 25/100\n",
      "195/195 [==============================] - 219s 1s/step - loss: 0.7412 - acc: 0.7872 - val_loss: 0.6126 - val_acc: 0.8299\n",
      "Epoch 26/100\n",
      "195/195 [==============================] - 214s 1s/step - loss: 0.7410 - acc: 0.7881 - val_loss: 0.6073 - val_acc: 0.8355\n",
      "Epoch 27/100\n",
      "195/195 [==============================] - 222s 1s/step - loss: 0.7307 - acc: 0.7904 - val_loss: 0.6251 - val_acc: 0.8300\n",
      "Epoch 28/100\n",
      "195/195 [==============================] - 217s 1s/step - loss: 0.7319 - acc: 0.7919 - val_loss: 0.6022 - val_acc: 0.8365\n",
      "Epoch 29/100\n",
      "195/195 [==============================] - 210s 1s/step - loss: 0.7299 - acc: 0.7932 - val_loss: 0.5875 - val_acc: 0.8391\n",
      "Epoch 30/100\n",
      "195/195 [==============================] - 208s 1s/step - loss: 0.7216 - acc: 0.7941 - val_loss: 0.6193 - val_acc: 0.8263\n",
      "Epoch 31/100\n",
      "195/195 [==============================] - 214s 1s/step - loss: 0.7252 - acc: 0.7922 - val_loss: 0.5998 - val_acc: 0.8319\n",
      "Epoch 32/100\n",
      "195/195 [==============================] - 211s 1s/step - loss: 0.7196 - acc: 0.7950 - val_loss: 0.6343 - val_acc: 0.8228\n",
      "Epoch 33/100\n",
      "195/195 [==============================] - 209s 1s/step - loss: 0.7243 - acc: 0.7930 - val_loss: 0.6213 - val_acc: 0.8297\n",
      "Epoch 34/100\n",
      "195/195 [==============================] - 208s 1s/step - loss: 0.7133 - acc: 0.7964 - val_loss: 0.6151 - val_acc: 0.8310\n",
      "\n",
      "Epoch 00034: ReduceLROnPlateau reducing learning rate to 0.25.\n",
      "Epoch 35/100\n",
      "195/195 [==============================] - 204s 1s/step - loss: 0.6853 - acc: 0.8057 - val_loss: 0.5690 - val_acc: 0.8448\n",
      "Epoch 36/100\n",
      "195/195 [==============================] - 205s 1s/step - loss: 0.6727 - acc: 0.8084 - val_loss: 0.5754 - val_acc: 0.8390\n",
      "Epoch 37/100\n",
      "195/195 [==============================] - 204s 1s/step - loss: 0.6671 - acc: 0.8094 - val_loss: 0.5709 - val_acc: 0.8431\n",
      "Epoch 38/100\n",
      "195/195 [==============================] - 205s 1s/step - loss: 0.6635 - acc: 0.8082 - val_loss: 0.5960 - val_acc: 0.8315\n",
      "Epoch 39/100\n",
      "195/195 [==============================] - 205s 1s/step - loss: 0.6559 - acc: 0.8100 - val_loss: 0.5550 - val_acc: 0.8449\n",
      "Epoch 40/100\n",
      "195/195 [==============================] - 200s 1s/step - loss: 0.6496 - acc: 0.8114 - val_loss: 0.5825 - val_acc: 0.8345\n",
      "Epoch 41/100\n",
      "195/195 [==============================] - 205s 1s/step - loss: 0.6483 - acc: 0.8110 - val_loss: 0.5635 - val_acc: 0.8402\n",
      "Epoch 42/100\n",
      "195/195 [==============================] - 254s 1s/step - loss: 0.6362 - acc: 0.8161 - val_loss: 0.5650 - val_acc: 0.8418\n",
      "Epoch 43/100\n",
      "195/195 [==============================] - 203s 1s/step - loss: 0.6374 - acc: 0.8133 - val_loss: 0.5699 - val_acc: 0.8387\n",
      "Epoch 44/100\n",
      "195/195 [==============================] - 203s 1s/step - loss: 0.6368 - acc: 0.8149 - val_loss: 0.5759 - val_acc: 0.8344\n",
      "\n",
      "Epoch 00044: ReduceLROnPlateau reducing learning rate to 0.125.\n",
      "Epoch 45/100\n",
      "195/195 [==============================] - 205s 1s/step - loss: 0.6220 - acc: 0.8180 - val_loss: 0.5417 - val_acc: 0.8457\n",
      "Epoch 46/100\n",
      "195/195 [==============================] - 206s 1s/step - loss: 0.6201 - acc: 0.8187 - val_loss: 0.5302 - val_acc: 0.8480\n",
      "Epoch 47/100\n",
      "195/195 [==============================] - 168s 861ms/step - loss: 0.6120 - acc: 0.8213 - val_loss: 0.5341 - val_acc: 0.8492\n",
      "Epoch 48/100\n",
      "195/195 [==============================] - 177s 909ms/step - loss: 0.6078 - acc: 0.8220 - val_loss: 0.5139 - val_acc: 0.8570\n",
      "Epoch 49/100\n",
      "195/195 [==============================] - 180s 921ms/step - loss: 0.6039 - acc: 0.8226 - val_loss: 0.5290 - val_acc: 0.8473\n",
      "Epoch 50/100\n",
      "195/195 [==============================] - 189s 969ms/step - loss: 0.5997 - acc: 0.8255 - val_loss: 0.5225 - val_acc: 0.8518\n",
      "Epoch 51/100\n",
      "195/195 [==============================] - 184s 944ms/step - loss: 0.6029 - acc: 0.8231 - val_loss: 0.5233 - val_acc: 0.8512\n",
      "Epoch 52/100\n",
      "195/195 [==============================] - 188s 966ms/step - loss: 0.5933 - acc: 0.8243 - val_loss: 0.5302 - val_acc: 0.8461\n",
      "Epoch 53/100\n",
      "195/195 [==============================] - 187s 960ms/step - loss: 0.5943 - acc: 0.8242 - val_loss: 0.5459 - val_acc: 0.8417\n",
      "\n",
      "Epoch 00053: ReduceLROnPlateau reducing learning rate to 0.0625.\n",
      "Epoch 54/100\n",
      "195/195 [==============================] - 187s 958ms/step - loss: 0.5821 - acc: 0.8276 - val_loss: 0.5153 - val_acc: 0.8514\n",
      "Epoch 55/100\n",
      "195/195 [==============================] - 193s 987ms/step - loss: 0.5848 - acc: 0.8263 - val_loss: 0.5169 - val_acc: 0.8509\n",
      "Epoch 56/100\n",
      "195/195 [==============================] - 180s 921ms/step - loss: 0.5848 - acc: 0.8267 - val_loss: 0.5189 - val_acc: 0.8491\n",
      "Epoch 57/100\n",
      "195/195 [==============================] - 188s 964ms/step - loss: 0.5796 - acc: 0.8281 - val_loss: 0.5075 - val_acc: 0.8536\n",
      "Epoch 58/100\n",
      "195/195 [==============================] - 189s 970ms/step - loss: 0.5776 - acc: 0.8303 - val_loss: 0.5072 - val_acc: 0.8557\n",
      "\n",
      "Epoch 00058: ReduceLROnPlateau reducing learning rate to 0.03125.\n",
      "Epoch 59/100\n",
      "195/195 [==============================] - 218s 1s/step - loss: 0.5768 - acc: 0.8294 - val_loss: 0.5119 - val_acc: 0.8528\n",
      "Epoch 60/100\n",
      "195/195 [==============================] - 197s 1s/step - loss: 0.5749 - acc: 0.8294 - val_loss: 0.5142 - val_acc: 0.8512\n",
      "Epoch 61/100\n",
      "195/195 [==============================] - 190s 976ms/step - loss: 0.5760 - acc: 0.8280 - val_loss: 0.5162 - val_acc: 0.8498\n",
      "Epoch 62/100\n",
      "195/195 [==============================] - 190s 975ms/step - loss: 0.5686 - acc: 0.8317 - val_loss: 0.5110 - val_acc: 0.8504\n",
      "Epoch 63/100\n",
      "195/195 [==============================] - 181s 926ms/step - loss: 0.5708 - acc: 0.8302 - val_loss: 0.5151 - val_acc: 0.8492\n",
      "\n",
      "Epoch 00063: ReduceLROnPlateau reducing learning rate to 0.015625.\n",
      "Epoch 64/100\n",
      "195/195 [==============================] - 170s 870ms/step - loss: 0.5679 - acc: 0.8333 - val_loss: 0.5090 - val_acc: 0.8513\n",
      "Epoch 65/100\n",
      "195/195 [==============================] - 215s 1s/step - loss: 0.5637 - acc: 0.8323 - val_loss: 0.5082 - val_acc: 0.8526\n",
      "Epoch 66/100\n",
      "195/195 [==============================] - 175s 898ms/step - loss: 0.5675 - acc: 0.8300 - val_loss: 0.5155 - val_acc: 0.8489\n",
      "Epoch 67/100\n",
      "195/195 [==============================] - 167s 855ms/step - loss: 0.5665 - acc: 0.8318 - val_loss: 0.5089 - val_acc: 0.8509\n",
      "Epoch 68/100\n",
      "195/195 [==============================] - 170s 871ms/step - loss: 0.5683 - acc: 0.8318 - val_loss: 0.5077 - val_acc: 0.8512\n",
      "\n",
      "Epoch 00068: ReduceLROnPlateau reducing learning rate to 0.0078125.\n",
      "Epoch 69/100\n",
      "195/195 [==============================] - 179s 917ms/step - loss: 0.5631 - acc: 0.8318 - val_loss: 0.5065 - val_acc: 0.8510\n",
      "Epoch 70/100\n",
      "195/195 [==============================] - 167s 858ms/step - loss: 0.5643 - acc: 0.8325 - val_loss: 0.5057 - val_acc: 0.8515\n",
      "Epoch 71/100\n",
      "195/195 [==============================] - 164s 839ms/step - loss: 0.5638 - acc: 0.8310 - val_loss: 0.5094 - val_acc: 0.8514\n",
      "Epoch 72/100\n",
      "195/195 [==============================] - 167s 855ms/step - loss: 0.5662 - acc: 0.8310 - val_loss: 0.5074 - val_acc: 0.8507\n",
      "Epoch 73/100\n",
      "195/195 [==============================] - 163s 837ms/step - loss: 0.5656 - acc: 0.8304 - val_loss: 0.5054 - val_acc: 0.8529\n",
      "\n",
      "Epoch 00073: ReduceLROnPlateau reducing learning rate to 0.00390625.\n",
      "Epoch 74/100\n",
      "195/195 [==============================] - 162s 830ms/step - loss: 0.5676 - acc: 0.8317 - val_loss: 0.5071 - val_acc: 0.8511\n",
      "Epoch 75/100\n",
      "195/195 [==============================] - 163s 838ms/step - loss: 0.5579 - acc: 0.8351 - val_loss: 0.5070 - val_acc: 0.8514\n",
      "Epoch 76/100\n",
      "195/195 [==============================] - 164s 843ms/step - loss: 0.5649 - acc: 0.8321 - val_loss: 0.5056 - val_acc: 0.8523\n",
      "Epoch 77/100\n",
      "195/195 [==============================] - 164s 840ms/step - loss: 0.5614 - acc: 0.8351 - val_loss: 0.5054 - val_acc: 0.8525\n",
      "Epoch 78/100\n",
      "195/195 [==============================] - 162s 833ms/step - loss: 0.5619 - acc: 0.8344 - val_loss: 0.5064 - val_acc: 0.8513\n",
      "\n",
      "Epoch 00078: ReduceLROnPlateau reducing learning rate to 0.001953125.\n",
      "Epoch 79/100\n",
      "195/195 [==============================] - 165s 844ms/step - loss: 0.5721 - acc: 0.8301 - val_loss: 0.5066 - val_acc: 0.8515\n",
      "Epoch 80/100\n",
      "195/195 [==============================] - 163s 838ms/step - loss: 0.5624 - acc: 0.8350 - val_loss: 0.5051 - val_acc: 0.8522\n",
      "Epoch 81/100\n",
      "195/195 [==============================] - 160s 820ms/step - loss: 0.5603 - acc: 0.8342 - val_loss: 0.5066 - val_acc: 0.8516\n",
      "Epoch 82/100\n",
      "195/195 [==============================] - 160s 823ms/step - loss: 0.5588 - acc: 0.8342 - val_loss: 0.5059 - val_acc: 0.8518\n",
      "Epoch 83/100\n",
      "195/195 [==============================] - 165s 847ms/step - loss: 0.5616 - acc: 0.8314 - val_loss: 0.5062 - val_acc: 0.8517\n",
      "\n",
      "Epoch 00083: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 84/100\n",
      "195/195 [==============================] - 203s 1s/step - loss: 0.5567 - acc: 0.8337 - val_loss: 0.5061 - val_acc: 0.8520\n",
      "Epoch 85/100\n",
      "195/195 [==============================] - 226s 1s/step - loss: 0.5634 - acc: 0.8346 - val_loss: 0.5061 - val_acc: 0.8518\n",
      "Epoch 86/100\n",
      "195/195 [==============================] - 216s 1s/step - loss: 0.5611 - acc: 0.8315 - val_loss: 0.5061 - val_acc: 0.8516\n",
      "Epoch 87/100\n",
      "195/195 [==============================] - 214s 1s/step - loss: 0.5603 - acc: 0.8341 - val_loss: 0.5049 - val_acc: 0.8523\n",
      "Epoch 88/100\n",
      "195/195 [==============================] - 209s 1s/step - loss: 0.5656 - acc: 0.8319 - val_loss: 0.5059 - val_acc: 0.8518\n",
      "\n",
      "Epoch 00088: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 89/100\n",
      "195/195 [==============================] - 202s 1s/step - loss: 0.5635 - acc: 0.8326 - val_loss: 0.5053 - val_acc: 0.8519\n",
      "Epoch 90/100\n",
      "195/195 [==============================] - 169s 864ms/step - loss: 0.5598 - acc: 0.8341 - val_loss: 0.5059 - val_acc: 0.8515\n",
      "Epoch 91/100\n",
      "195/195 [==============================] - 188s 965ms/step - loss: 0.5650 - acc: 0.8319 - val_loss: 0.5064 - val_acc: 0.8515\n",
      "Epoch 92/100\n",
      "195/195 [==============================] - 172s 880ms/step - loss: 0.5614 - acc: 0.8321 - val_loss: 0.5067 - val_acc: 0.8514\n",
      "Epoch 93/100\n",
      "195/195 [==============================] - 181s 927ms/step - loss: 0.5661 - acc: 0.8311 - val_loss: 0.5069 - val_acc: 0.8515\n",
      "\n",
      "Epoch 00093: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 94/100\n",
      "195/195 [==============================] - 178s 913ms/step - loss: 0.5608 - acc: 0.8327 - val_loss: 0.5055 - val_acc: 0.8518\n",
      "Epoch 95/100\n",
      "195/195 [==============================] - 164s 841ms/step - loss: 0.5548 - acc: 0.8353 - val_loss: 0.5052 - val_acc: 0.8524\n",
      "Epoch 96/100\n",
      "195/195 [==============================] - 170s 870ms/step - loss: 0.5655 - acc: 0.8303 - val_loss: 0.5055 - val_acc: 0.8525\n",
      "Epoch 97/100\n",
      "195/195 [==============================] - 168s 861ms/step - loss: 0.5575 - acc: 0.8360 - val_loss: 0.5060 - val_acc: 0.8522\n",
      "Epoch 98/100\n",
      "195/195 [==============================] - 196s 1s/step - loss: 0.5628 - acc: 0.8312 - val_loss: 0.5059 - val_acc: 0.8523\n",
      "\n",
      "Epoch 00098: ReduceLROnPlateau reducing learning rate to 0.001.\n",
      "Epoch 99/100\n",
      "195/195 [==============================] - 176s 903ms/step - loss: 0.5603 - acc: 0.8333 - val_loss: 0.5057 - val_acc: 0.8521\n",
      "Epoch 100/100\n",
      "195/195 [==============================] - 167s 858ms/step - loss: 0.5601 - acc: 0.8335 - val_loss: 0.5050 - val_acc: 0.8519\n"
     ]
    }
   ],
   "source": [
    "# Define hyperparameters\n",
    "batch_size = 256  # size of mini-baches\n",
    "epochs = 100      # how many times all traing examples will be used to train this model further\n",
    "\n",
    "# Compile the model using optimizer\n",
    "model3.compile(loss=keras.losses.categorical_crossentropy,\n",
    "               optimizer=keras.optimizers.Adadelta(),\n",
    "               metrics=['acc']) # List of metrics to be evaluated by the model during training and validating: https://keras.io/models/model/#compile\n",
    "\n",
    "# Learning rate reduction durint the training process: https://keras.io/callbacks/#reducelronplateau\n",
    "learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', # quantity to be monitored (val_loss)\n",
    "                                            factor=0.5, # factor by which the learning rate will be reduced. new_lr = lr * factor\n",
    "                                            patience=5, # number of epochs that produced the monitored quantity with no improvement after which training will be stopped\n",
    "                                            verbose=1,  # 0: quiet, 1: update messages.\n",
    "                                            min_lr=0.001) # lower bound on the learning rate\n",
    "\n",
    "# Augmentation of training data. It generates batches of tensor image data with real-time data augmentation. The data will be looped over in batches.\n",
    "datagen = ImageDataGenerator(\n",
    "          rotation_range=20,       # rotate images in degrees up to the given degrees\n",
    "          width_shift_range=0.1,   # shift images horizontally\n",
    "          height_shift_range=0.1,  # shift images vertically\n",
    "          horizontal_flip=True)    # flip images (left<->right)\n",
    "# Computes the internal data stats related to the data-dependent transformations, based on an array of samples x_train\n",
    "datagen.fit(x_train)\n",
    "\n",
    "# Train the model, validate, evaluate, and present scores\n",
    "history=model3.fit_generator(datagen.flow(x_train, y_train, batch_size=batch_size),\n",
    "                            epochs=epochs,\n",
    "                            steps_per_epoch=x_train.shape[0]//batch_size,  # no of mini-batches \n",
    "                            validation_data=(x_val, y_val),\n",
    "                            verbose=1,\n",
    "                            callbacks=[learning_rate_reduction])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save weights of the model before training to enable to start training it again\n",
    "model3.save(models_dir + 'CNN_CIFAR-10_2.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loss: 0.379984456281662\n",
      "Train accuracy: 0.896619975566864\n",
      "Bias: 0.896619975566864\n",
      "Validation loss: 0.5050123132705688\n",
      "Validation accuracy: 0.8518999814987183\n",
      "Variance: -0.04471999406814575\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 41060 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 36976 characters]\n",
      "text/plain": [
       "<Figure size 1080x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "score_train = model3.evaluate(x_train, y_train, verbose=0)\n",
    "print('Train loss:', score_train[0])\n",
    "print('Train accuracy:', score_train[1])\n",
    "print('Bias:', score_train[1])   # Assuming that the Bayes optimal error is equal to 0.\n",
    "\n",
    "score_val = model3.evaluate(x_val, y_val, verbose=0)\n",
    "print('Validation loss:', score_val[0])\n",
    "print('Validation accuracy:', score_val[1])\n",
    "print('Variance:', score_val[1] - score_train[1])\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (15.0, 5.0) # set default size of plots\n",
    "\n",
    "# Plot training & validation accuracy values: https://keras.io/visualization/#training-history-visualization\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('Model accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(['Train', 'Val'], loc='upper left')  # OR plt.legend(['Train', 'Validation'], loc='upper left')\n",
    "plt.show()\n",
    "\n",
    "# Plot training & validation loss values: https://keras.io/visualization/#training-history-visualizatio\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('Model loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend(['Train', 'Val'], loc='lower left')  # OR plt.legend(['Train', 'Validation'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42640 characters]\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      0.88      0.87      1000\n",
      "           1       0.94      0.94      0.94      1000\n",
      "           2       0.87      0.73      0.80      1000\n",
      "           3       0.79      0.63      0.70      1000\n",
      "           4       0.82      0.86      0.84      1000\n",
      "           5       0.85      0.73      0.79      1000\n",
      "           6       0.76      0.96      0.85      1000\n",
      "           7       0.87      0.91      0.89      1000\n",
      "           8       0.91      0.93      0.92      1000\n",
      "           9       0.85      0.94      0.90      1000\n",
      "\n",
      "    accuracy                           0.85     10000\n",
      "   macro avg       0.85      0.85      0.85     10000\n",
      "weighted avg       0.85      0.85      0.85     10000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Use the trained model for predictions of the val data (in this case, we use validation data as val data)\n",
    "y_pred_val = model3.predict(x_val)\n",
    "\n",
    "# Take the class with the highest probability from the val predictions as a winning one\n",
    "max_y_pred_val = np.argmax(y_pred_val, axis=1)\n",
    "max_y_val = np.argmax(y_val, axis=1)\n",
    "\n",
    "# Show the confution matrix of the collected results\n",
    "show_confusion_matrix(max_y_val, max_y_pred_val, num_classes)\n",
    "\n",
    "# Print classification report\n",
    "print(classification_report(max_y_val, max_y_pred_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of misclassified examples:  1481\n",
      "Misclassified examples:\n",
      "[   3   24   32 ... 9982 9985 9989]\n",
      "Misclassified images (original class : predicted class):\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 6310496 characters]\n",
      "text/plain": [
       "<Figure size 720x21456 with 1481 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Find out misclassified examples\n",
    "classcheck = max_y_val - max_y_pred_val  # 0 - when the class is the same, 1 - otherwise\n",
    "misclassified = np.where(classcheck != 0)[0]\n",
    "num_misclassified = len(misclassified)\n",
    "\n",
    "# Print misclassification report\n",
    "print('Number of misclassified examples: ', str(num_misclassified))\n",
    "print('Misclassified examples:')\n",
    "print(misclassified)\n",
    "\n",
    "# Show misclassified examples:\n",
    "print('Misclassified images (original class : predicted class):')\n",
    "plt.rcParams['figure.figsize'] = (2.5, 2.5) # set default size of plots\n",
    "col = 10\n",
    "row = 2 * math.ceil(num_misclassified / col)\n",
    "fig = plt.figure(figsize=(col, row))\n",
    "for index in range(0,num_misclassified):\n",
    "    fig.add_subplot(row, col, index + 1 + col*(index//col))\n",
    "    plt.axis('off')\n",
    "    plt.imshow(x_val[misclassified[index]]) # index of the validation sample picture\n",
    "    plt.title(str(max_y_val[misclassified[index]]) + \":\" + str(max_y_pred_val[misclassified[index]]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After 100 epochs, we increased the model accuracy from 0.81 to 0.85, so the number of incorrectly classified validation examples dropped about a few hundred. Our model is still not perfect. It's time for you! What do you propose to do with this model to perform better than the presented model? Implement your ideas in the following assignments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ASSIGNMENTS\n",
    "\n",
    "* Copy and paste the above-developed model to the cells below.\n",
    "* Modify hyperparameters of the model (the structure, the number of layers in the network and the number of filters per convolution layer, dropout parameters etc.) and achieve better accuracy for both models: MNIST and CIFAR-10.\n",
    "* Reduce the high bias of the CIFAR-10 model, not raising or even still decreasing the model variance.\n",
    "* Compare accuracy and loss of your model with the previously trained models (PlotTwoAccuracyComparison and PlotTwoLossComparison).\n",
    "* Try to perform similar experiments on a different training dataset of images, e.g. downloaded from <a href=\"www.kaggle.com\">Kaggle.com</a> or a different source. Create a new CNN model and implement regularizations and other optimization methods trying to decrease bias and variance of the model. \n",
    "\n",
    "* Send your developed CNN model as an answer to this assignment on MS Teams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
