A01_ae_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autoencoder | Digit Recognizer\n",
    "This is a refresh on the `autoencoder` algorithm, the autoencoder algorithms have limitations due to the fact that the \"internal representation\" achieved at the end of the __encoding__ part, _does not follow any known distribution representation_. Hence we can not use it to generate proper new __similar__ (to the trained data). __Autoencoders__ are used for image _denoizing_.\n",
    "\n",
    "## Tutorials:\n",
    "Tutorials employed as `inspiration or base` either to understand better the __concept__ or to use as a starting point for doing the _code implementation_. \n",
    " - [Building Autoencoders in Keras](https://blog.keras.io/building-autoencoders-in-keras.html)\n",
    " - [Introduction to autoencoders](https://www.jeremyjordan.me/autoencoders/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1| Base\n",
    "\n",
    "![Autoencoder](notes/theory_autoencoder.png)\n",
    "\n",
    "The __bottleneck__ learns a _dimentionally reduced_ representation of the __input__ in order to be able to reconstruct it in the __output__ with as few _loss_, quality loss, as possible.\n",
    "\n",
    "\n",
    "#### Results\n",
    "We can observe a plateau, by observing both the __accuracy__ and __loss__ graphics, reaching by the epoch number _80_, having both __train/test__ accuracy with similar values.\n",
    "\n",
    "![01_Base_accuracy](results/A01_ae_mnist/1_Base_accuracy.png)\n",
    "\n",
    "![01_Base_loss](results/A01_ae_mnist/1_Base_loss.png)\n",
    "\n",
    "As for a more __visual__ observation of the neural network improvements: \n",
    "![01_Base_comparison_initial](results/A01_ae_mnist/1_Base_comparison_initial.png)\n",
    "\n",
    "Between the __first__ iterations and the __last__ one we do not appreciate an _extreme_ (subjective). improvement, though there is an _increase_ in the __nitidity__ of the generated numbers.\n",
    "\n",
    "![01_Base_comparison_final](results/A01_ae_mnist/1_Base_comparison_final.png)\n",
    "\n",
    "##### Manifold distribution representation\n",
    "We can observe the distribution of the different values in the first _two dimentions_ of the space. The pattern that clearly arrises is that __there is no clear separation between data points _position_ and _value_ .__.\n",
    "\n",
    "![01_Base_distribution_generator](results/A01_ae_mnist/1_Base_distribution_generator.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2| Sparse\n",
    "The introduction of sparsity helps to prevent the _neural network_ of fully memorizing the __dataset__. This is achieved by:\n",
    " - Using __regularization__:\n",
    "    - L1 regularization: to create sparse set of weigths\n",
    "    - L2 regularization: to reduce the overall value of the weights\n",
    " - Using __dropout__: even if this technique is not _directly_ recommended on the `introduction to autoencoders`, it is a standar thecnique to increase the __redundancy__ of the network, by forcing it to create new _activation paths_.\n",
    " \n",
    "\n",
    "#### Results\n",
    "\n",
    " The results shown, are applying a combination of `regularization (L1) and dropout`\n",
    " \n",
    "![02_Sparse_accuracy](results/A01_ae_mnist/2_Sparse_accuracy.png)\n",
    "\n",
    "![02_Sparse_loss](results/A01_ae_mnist/2_Sparse_loss.png)\n",
    "\n",
    "Results of the model once completely trained:\n",
    "![2_Sparse_comparison_final](results/A01_ae_mnist/2_Sparse_comparison_final.png)\n",
    "\n",
    "##### Manifold distribution representation\n",
    "We can observe the distribution of the different values in the first _two dimentions_ of the space. The pattern that clearly arrises is that __there is no clear separation between data points _position_ and _value_ .__.\n",
    "\n",
    "![2_Sparse_distribution_generator](results/A01_ae_mnist/2_Sparse_distribution_generator.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "The implementation of the __autoenconders algorithms__ has shown the _utility_ of them, there are obviously not focused to be used as a standar compression tool, as they need to be trained on tailored datasets. Though it is possible to appreciate the _ability_ to reconstruct simple images, showed by the algorithm.\n",
    "\n",
    "As for the difference between the __base__ and the __sparse__ models, no real difference could be appreciated. Neither on the _accuracy/loss_ side, nor on the more __visual__ part by looking at the generated output. In any case the inclusion of regularization, tends to increase the robustness of the model, so it is _recommended_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kaggle\n",
    "### [Denoising Dirty Documents](https://www.kaggle.com/c/denoising-dirty-documents)\n",
    "This dataset will be used to practise the concepts learned during the `tutorials`\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}