{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tut5 - Autoencoder_dim_Reduction.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWgXQGVkE8gs"
      },
      "source": [
        "#Autoencoder With Neural Networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hWuiOjfVbuMS"
      },
      "source": [
        "\"Autoencoding\" is a data compression algorithm where the compression and decompression functions are 1) data-specific, 2) lossy, and 3) learned automatically from examples rather than engineered by a human. Additionally, in almost all contexts where the term \"autoencoder\" is used, the compression and decompression functions are implemented with unsupervised neural networks (i.e., no class labels or labeled data).\n",
        "\n",
        "1) Autoencoders are data-specific, which means that they will only be able to compress data similar to what they have been trained on. This is different from, say, the MPEG-2 Audio Layer III (MP3) compression algorithm, which only holds assumptions about \"sound\" in general, but not about specific types of sounds.An autoencoder trained on pictures of faces would do a rather poor job of compressing pictures of trees, because the features it would learn would be face-specific.\n",
        "\n",
        "2) Autoencoders are lossy, which means that the decompressed outputs will be degraded compared to the original inputs \n",
        "\n",
        "3) Autoencoders are learned automatically from data examples, which is a useful property: it means that it is easy to train specialized instances of the algorithm that will perform well on a specific type of input. It doesn't require any new engineering, just appropriate training data.\n",
        "\n",
        "To build an autoencoder, you need three things: an encoding function, a decoding function, and a distance function between the amount of information loss between the compressed representation of your data and the decompressed representation (i.e. a \"loss\" function). \n",
        "\n",
        "Autoencoders are typically used for:\n",
        "\n",
        "* Dimensionality reduction (i.e., think PCA but more powerful/intelligent).\n",
        "* Denoising (ex., removing noise and preprocessing images to improve OCR accuracy)\n",
        "* Anomaly/outlier detection (ex., detecting mislabeled data points in a dataset or detecting when an input data point falls well outside our typical data distribution).\n",
        "\n",
        "Today two interesting practical applications of autoencoders are data denoising (which we feature later in this post), and dimensionality reduction for data visualization. With appropriate dimensionality and sparsity constraints, autoencoders can learn data projections that are more interesting than PCA or other basic techniques.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VvUaFwhKuGfA"
      },
      "source": [
        "![alt text](https://www.pyimagesearch.com/wp-content/uploads/2020/02/keras_autoencoder_arch_flow.png)\n",
        "\n",
        "We input a digit to the autoencoder.\n",
        "The encoder subnetwork creates a latent representation of the digit. This latent representation is substantially smaller (in terms of dimensionality) than the input.\n",
        "The decoder subnetwork then reconstructs the original digit from the latent representation.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q4SjYTQ7twvN"
      },
      "source": [
        "#Import libraries for neural network, numpy for matrix operations, and matplotlib for visualization\n",
        "#Keras library is a high level API for deep leanring and neural networks\n",
        "\n",
        "from keras.datasets import mnist\n",
        "from keras.layers import Input, Dense\n",
        "from keras.models import Model\n",
        "from keras import backend as b\n",
        "from keras import initializers,regularizers\n",
        "from keras.optimizers import SGD,Adam\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21lvKZ_JjO7F"
      },
      "source": [
        "Loading the MNIST dataset images and not their labels. We want to reconstruct the images as output of the autoencoder and hence we do not need labels.\n",
        "Creating a training set and test set and normalizing the data for better training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uvd4dY5UtA1Y"
      },
      "source": [
        "(X_train, _), (X_test, _) = mnist.load_data()\n",
        "\n",
        "X_train = X_train.astype('float32')/255.\n",
        "X_test = X_test.astype('float32')/255.\n",
        "\n",
        "X_train = X_train.reshape(len(X_train), np.prod(X_train.shape[1:]))\n",
        "X_test = X_test.reshape(len(X_test), np.prod(X_test.shape[1:]))\n",
        "print(X_train.shape)\n",
        "print(X_test.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oR1YooN8tHAv"
      },
      "source": [
        "Our input image is a 2D matrix of size 28x28. When we represented it as a vector it has a dimension of 784. We will encode it to 32 and the decode it back to the original dimension of 784\n",
        "\n",
        "![alt text](https://miro.medium.com/max/806/1*Widu_pWRjbGvbbmZ5IiT4Q.png)\n",
        "\n",
        "Encoded image will have a compression rate of 784/32=24.5\n",
        "\n",
        "![alt text](https://wiki.pathmind.com/images/wiki/deep_autoencoder.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1XuZrRNftVaI"
      },
      "source": [
        "b.clear_session()\n",
        "#We need to take the input image of dimension 784 and convert it to keras tensors.\n",
        "input_img= Input(shape=(784,))\n",
        "#We now create the encoder and the decoder based on the figure above.\n",
        "#Input image will be Encoded to 32 units and the activation will be relu and the input is the tensors of input image\n",
        "encoded = Dense(units=32, activation='relu')(input_img)\n",
        "#Decoder will have 784 units as it needs to reconstruct the input image back to its original dimension.\n",
        "#As decoded is the output layer of the autoencoder we will use sigmoid activation function. encoded image will be the input to the decoder\n",
        "#Sigmoid allows us to model the output as the presence of a white pixel or not. \n",
        "decoded = Dense(units=784, activation='sigmoid')(encoded)\n",
        "#We now create the autoencoder with input as the input image and output as the decoder\n",
        "autoencoder=Model(input_img, decoded)\n",
        "autoencoder.summary()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ilPPaUcO7kri"
      },
      "source": [
        "We train the autoencoder end to end, which has both the encoder and decoder parts. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9xzXUur6vERT"
      },
      "source": [
        "#We can also extract the encoder which takes input as input images and the output is the encoded image.\n",
        "encoder = Model(input_img, encoded)\n",
        "#let see the structure of the encoded model\n",
        "encoder.summary()\n",
        "#We now compile the autoencoder model with adadelta optimizer. \n",
        "#As pixels have a value of 0 or 1 we use binary_crossentropy as the loss function and the metrics used for the performance of the model is accuracy\n",
        "#i.e., the number of pixels that are correctly predicted. Alternativelly, we can think of each output node as the probability of having a white pixel. \n",
        "autoencoder.compile(optimizer='adam', loss='binary_crossentropy', metrics=['binary_accuracy'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YzUemFpxvXre"
      },
      "source": [
        "#We now train the autoencoder using the training data with 20 epochs and batch size of 256. We also shuffle the training data\n",
        "autoencoder.fit(X_train, X_train,\n",
        "                epochs=20,\n",
        "                batch_size=256,\n",
        "                shuffle=True,\n",
        "                validation_data=(X_test, X_test)\n",
        "                )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9KoJbj27zjZ"
      },
      "source": [
        "During training it is important to look at the loss. It should decrease. Similarly we should observe that the accuracy has increased.\n",
        "Finally, you should also keep an eye at validation losses and accuracies that they follow the same trends to identify overfitting."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G2jceflI_qah"
      },
      "source": [
        "#Visualizing the Encoding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L2wcvmW9kSpw"
      },
      "source": [
        "Predicting the test set. We want to view the encoded images as well as the reconstructed images so we fit the test data on both autoencoder as well as encoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3YYPeE-9vcbx"
      },
      "source": [
        "encoded_imgs = encoder.predict(X_test)\n",
        "predicted = autoencoder.predict(X_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DgK4K9kTkMqE"
      },
      "source": [
        "Let’s plot the original input, encoded images and the reconstructed images using matplotlib\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fwETQ4Fivkhf"
      },
      "source": [
        "plt.figure(figsize=(40, 4))\n",
        "for i in range(10):\n",
        "    # display original\n",
        "    ax = plt.subplot(3, 20, i + 1)\n",
        "    plt.imshow(X_test[i].reshape(28, 28))\n",
        "    plt.gray()\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)\n",
        "    \n",
        "    # display encoded image\n",
        "    ax = plt.subplot(3, 20, i + 1 + 20)\n",
        "    plt.imshow(encoded_imgs[i].reshape(8,4))\n",
        "    plt.gray()\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)\n",
        "    # display reconstruction\n",
        "    ax = plt.subplot(3, 20, 2*20 +i+ 1)\n",
        "    plt.imshow(predicted[i].reshape(28, 28))\n",
        "    plt.gray()\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)\n",
        "    \n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KBk9X8T6kwQJ"
      },
      "source": [
        "Notice that the outputs are similar with the input. Furthermore, encodings of similar numbers seem to share some patterns. The outputs are fuzzy though. We could try a deeper network or train for longer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3X04pLxg870a"
      },
      "source": [
        "The representations learned by autoencoders can be used for downstream tasks such as classification or regression. Furthermore, autoencoder variants provide ways to model and learn distributions so that they can be used as generative models (e.g.,  Variational autoencoders (VAEs))."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MPJWPT7S9Ubp"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}