tfwss/model_train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \"Simple Does It\" Grabcut Training for Instance Segmentation\n",
    "\n",
    "This notebook performs training of the SDI Grabcut weakly supervised model for **instance segmentation**. Following the instructions provided in Section \"6. Instance Segmentation Results\" of the \"Simple Does It\" paper, we use the Berkeley-augmented Pascal VOC segmentation dataset that provides per-instance segmentation masks for VOC2012 data.\n",
    "\n",
    "The Berkley augmented dataset can be downloaded from [here](\n",
    "http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz)\n",
    "\n",
    "The SDI Grabcut training is done using a **4-channel input** VGG16 network pre-trained on ImageNet, so make sure to run the [`VGG16 Net Surgery`](net_surgery.ipynb) notebook first!\n",
    "\n",
    "To monitor training, run:\n",
    "```\n",
    "# On Windows\n",
    "tensorboard --logdir E:\\repos\\tf-wss\\tfwss\\models\\vgg_16_4chan_weak\n",
    "# On Ubuntu\n",
    "tensorboard --logdir /media/EDrive/repos/tf-wss/tfwss/models/vgg_16_4chan_weak\n",
    "http://<hostname>:6006\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\"\n",
    "model_train.ipynb\n",
    "\n",
    "SDI Grabcut weakly supervised model trainer (instance segmentation)\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Based on:\n",
    "  - https://github.com/scaelles/OSVOS-TensorFlow/blob/master/osvos_parent_demo.py\n",
    "    Written by Sergi Caelles (scaelles@vision.ee.ethz.ch)\n",
    "    This file is part of the OSVOS paper presented in:\n",
    "      Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixe, Daniel Cremers, Luc Van Gool\n",
    "      One-Shot Video Object Segmentation\n",
    "      CVPR 2017\n",
    "    Unknown code license\n",
    "\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import sys\n",
    "import tensorflow as tf\n",
    "slim = tf.contrib.slim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import model files\n",
    "import model\n",
    "from dataset import BKVOCDataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model paths\n",
    "# Pre-trained VGG_16 downloaded from http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz\n",
    "imagenet_ckpt = 'models/vgg_16_4chan/vgg_16_4chan.ckpt'\n",
    "segnet_stream = 'weak'\n",
    "ckpt_name = 'vgg_16_4chan_' + segnet_stream\n",
    "logs_path = 'models/' + ckpt_name\n",
    "\n",
    "# Training parameters\n",
    "gpu_id = 0\n",
    "iter_mean_grad = 10\n",
    "max_training_iters_1 = 15000\n",
    "max_training_iters_2 = 30000\n",
    "max_training_iters_3 = 50000\n",
    "save_step = 5000\n",
    "test_image = None\n",
    "display_step = 100\n",
    "ini_lr = 1e-8\n",
    "boundaries = [10000, 15000, 25000, 30000, 40000]\n",
    "values = [ini_lr, ini_lr * 0.1, ini_lr, ini_lr * 0.1, ini_lr, ini_lr * 0.1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Load the Berkeley-augmented Pascal VOC 2012 segmentation dataset\n",
    "if sys.platform.startswith(\"win\"):\n",
    "    dataset_root = \"E:/datasets/bk-voc/benchmark_RELEASE/dataset\"\n",
    "else:\n",
    "    dataset_root = '/media/EDrive/datasets/bk-voc/benchmark_RELEASE/dataset'\n",
    "dataset = BKVOCDataset(phase='train', dataset_root=dataset_root)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Configuration:\n",
      "  in_memory            False\n",
      "  data_aug             False\n",
      "  use_cache            False\n",
      "  use_grabcut_labels   True\n",
      "  phase                train\n",
      "  samples              20172\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "dataset.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offline Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = weak/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = weak/conv1/conv1_1/weights:0, shape = (3, 3, 4, 64)\n",
      "   name = weak/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = weak/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = weak/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = weak/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = weak/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = weak/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = weak/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = weak/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = weak/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = weak/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = weak/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = weak/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = weak/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = weak/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = weak/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = weak/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = weak/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = weak/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = weak/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = weak/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = weak/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = weak/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = weak/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = weak/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = weak/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from pre-trained imagenet model...\n",
      "INFO:tensorflow:Restoring parameters from models/vgg_16_4chan/vgg_16_4chan.ckpt\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-02-09 20:25:18.684456 Iter 100: Training Loss = 4442.2686\n",
      "2018-02-09 20:28:56.269373 Iter 200: Training Loss = 676.8325\n",
      "2018-02-09 20:32:27.232507 Iter 300: Training Loss = 37543.4336\n",
      "2018-02-09 20:35:45.813327 Iter 400: Training Loss = 455.7260\n",
      "2018-02-09 20:39:05.004978 Iter 500: Training Loss = 45564.7812\n",
      "2018-02-09 20:42:25.434892 Iter 600: Training Loss = 0.4069\n",
      "2018-02-09 20:45:39.067948 Iter 700: Training Loss = 0.4069\n",
      "2018-02-09 20:48:55.265078 Iter 800: Training Loss = 2906.9341\n",
      "2018-02-09 20:52:07.652900 Iter 900: Training Loss = 104.8880\n",
      "2018-02-09 20:55:18.022249 Iter 1000: Training Loss = 180998.6250\n",
      "2018-02-09 20:58:28.321771 Iter 1100: Training Loss = 21009.3906\n",
      "2018-02-09 21:01:38.254535 Iter 1200: Training Loss = 519.9033\n",
      "2018-02-09 21:04:38.210445 Iter 1300: Training Loss = 3984.0244\n",
      "2018-02-09 21:07:35.524935 Iter 1400: Training Loss = 22411.4766\n",
      "2018-02-09 21:10:29.827615 Iter 1500: Training Loss = 0.4069\n",
      "2018-02-09 21:13:23.518550 Iter 1600: Training Loss = 3257.2742\n",
      "2018-02-09 21:16:15.471382 Iter 1700: Training Loss = 184404.5156\n",
      "2018-02-09 21:19:09.709009 Iter 1800: Training Loss = 30457.8477\n",
      "2018-02-09 21:22:02.582099 Iter 1900: Training Loss = 1597.2654\n",
      "2018-02-09 21:24:55.309265 Iter 2000: Training Loss = 9105.4873\n",
      "2018-02-09 21:27:46.258357 Iter 2100: Training Loss = 109754.7812\n",
      "2018-02-09 21:30:35.098105 Iter 2200: Training Loss = 93004.5703\n",
      "2018-02-09 21:33:24.062814 Iter 2300: Training Loss = 8245.0264\n",
      "2018-02-09 21:36:13.301680 Iter 2400: Training Loss = 0.4069\n",
      "2018-02-09 21:39:02.516252 Iter 2500: Training Loss = 10547.9775\n",
      "2018-02-09 21:41:51.676649 Iter 2600: Training Loss = 40651.1406\n",
      "2018-02-09 21:44:41.364199 Iter 2700: Training Loss = 11088.3955\n",
      "2018-02-09 21:47:29.297514 Iter 2800: Training Loss = 782.4946\n",
      "2018-02-09 21:50:18.628700 Iter 2900: Training Loss = 91149.9609\n",
      "2018-02-09 21:53:08.045233 Iter 3000: Training Loss = 11837.8682\n",
      "2018-02-09 21:55:56.900555 Iter 3100: Training Loss = 70106.5625\n",
      "2018-02-09 21:58:45.665963 Iter 3200: Training Loss = 173926.3906\n",
      "2018-02-09 22:01:35.033844 Iter 3300: Training Loss = 134847.3125\n",
      "2018-02-09 22:04:23.415975 Iter 3400: Training Loss = 0.4069\n",
      "2018-02-09 22:07:12.031294 Iter 3500: Training Loss = 125094.2812\n",
      "2018-02-09 22:10:01.593270 Iter 3600: Training Loss = 266118.4375\n",
      "2018-02-09 22:12:49.798849 Iter 3700: Training Loss = 4760.7700\n",
      "2018-02-09 22:15:39.372377 Iter 3800: Training Loss = 182.9610\n",
      "2018-02-09 22:18:28.327406 Iter 3900: Training Loss = 3934.8997\n",
      "2018-02-09 22:21:16.078285 Iter 4000: Training Loss = 12297.1963\n",
      "2018-02-09 22:24:05.555932 Iter 4100: Training Loss = 2378.7810\n",
      "2018-02-09 22:26:53.474285 Iter 4200: Training Loss = 75285.6719\n",
      "2018-02-09 22:29:42.635403 Iter 4300: Training Loss = 119119.8516\n",
      "2018-02-09 22:32:32.642690 Iter 4400: Training Loss = 0.4070\n",
      "2018-02-09 22:35:20.936090 Iter 4500: Training Loss = 46869.3594\n",
      "2018-02-09 22:38:09.190423 Iter 4600: Training Loss = 0.4070\n",
      "2018-02-09 22:40:57.507408 Iter 4700: Training Loss = 637.3744\n",
      "2018-02-09 22:43:45.883475 Iter 4800: Training Loss = 0.4070\n",
      "2018-02-09 22:46:34.647568 Iter 4900: Training Loss = 173.0435\n",
      "2018-02-09 22:49:22.456440 Iter 5000: Training Loss = 10143.7344\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-5000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-5000\n",
      "2018-02-09 22:52:14.795808 Iter 5100: Training Loss = 0.4070\n",
      "2018-02-09 22:55:03.582199 Iter 5200: Training Loss = 19309.3828\n",
      "2018-02-09 22:57:52.577130 Iter 5300: Training Loss = 544.0726\n",
      "2018-02-09 23:00:41.156702 Iter 5400: Training Loss = 0.4070\n",
      "2018-02-09 23:03:30.017770 Iter 5500: Training Loss = 80.9669\n",
      "2018-02-09 23:06:19.647981 Iter 5600: Training Loss = 335.9449\n",
      "2018-02-09 23:09:08.758425 Iter 5700: Training Loss = 59641.8633\n",
      "2018-02-09 23:11:58.513345 Iter 5800: Training Loss = 171425.7812\n",
      "2018-02-09 23:14:47.071905 Iter 5900: Training Loss = 2767.3337\n",
      "2018-02-09 23:17:36.156909 Iter 6000: Training Loss = 7.6117\n",
      "2018-02-09 23:20:25.887765 Iter 6100: Training Loss = 106352.6484\n",
      "2018-02-09 23:23:14.668094 Iter 6200: Training Loss = 0.4070\n",
      "2018-02-09 23:26:02.692860 Iter 6300: Training Loss = 0.4070\n",
      "2018-02-09 23:28:50.674973 Iter 6400: Training Loss = 30665.5371\n",
      "2018-02-09 23:31:39.973952 Iter 6500: Training Loss = 9510.2207\n",
      "2018-02-09 23:34:28.599843 Iter 6600: Training Loss = 0.4070\n",
      "2018-02-09 23:37:15.833248 Iter 6700: Training Loss = 0.4070\n",
      "2018-02-09 23:40:04.774288 Iter 6800: Training Loss = 74766.8125\n",
      "2018-02-09 23:42:53.371047 Iter 6900: Training Loss = 24086.0391\n",
      "2018-02-09 23:45:41.979809 Iter 7000: Training Loss = 165.0621\n",
      "2018-02-09 23:48:29.771407 Iter 7100: Training Loss = 204973.3438\n",
      "2018-02-09 23:51:18.000323 Iter 7200: Training Loss = 12810.8193\n",
      "2018-02-09 23:54:06.935002 Iter 7300: Training Loss = 61.7052\n",
      "2018-02-09 23:56:56.144143 Iter 7400: Training Loss = 4369.4199\n",
      "2018-02-09 23:59:44.618024 Iter 7500: Training Loss = 17581.0352\n",
      "2018-02-10 00:02:34.121833 Iter 7600: Training Loss = 106.9791\n",
      "2018-02-10 00:05:22.613567 Iter 7700: Training Loss = 369261.8750\n",
      "2018-02-10 00:08:11.761375 Iter 7800: Training Loss = 0.4070\n",
      "2018-02-10 00:11:01.066725 Iter 7900: Training Loss = 3331.7039\n",
      "2018-02-10 00:13:50.337052 Iter 8000: Training Loss = 7232.5957\n",
      "2018-02-10 00:16:39.651164 Iter 8100: Training Loss = 32806.7031\n",
      "2018-02-10 00:19:27.909643 Iter 8200: Training Loss = 653.2996\n",
      "2018-02-10 00:22:17.345540 Iter 8300: Training Loss = 40391.6680\n",
      "2018-02-10 00:25:06.825883 Iter 8400: Training Loss = 227.2575\n",
      "2018-02-10 00:27:54.916882 Iter 8500: Training Loss = 339.7477\n",
      "2018-02-10 00:30:43.231766 Iter 8600: Training Loss = 1603.3856\n",
      "2018-02-10 00:33:31.624866 Iter 8700: Training Loss = 48702.2656\n",
      "2018-02-10 00:36:20.294366 Iter 8800: Training Loss = 217207.1719\n",
      "2018-02-10 00:39:09.045920 Iter 8900: Training Loss = 0.4070\n",
      "2018-02-10 00:41:57.235246 Iter 9000: Training Loss = 11338.2061\n",
      "2018-02-10 00:44:46.556868 Iter 9100: Training Loss = 4712.5269\n",
      "2018-02-10 00:47:35.711322 Iter 9200: Training Loss = 14396.8936\n",
      "2018-02-10 00:50:24.145204 Iter 9300: Training Loss = 2782.0564\n",
      "2018-02-10 00:53:12.352812 Iter 9400: Training Loss = 10947.9121\n",
      "2018-02-10 00:56:00.057794 Iter 9500: Training Loss = 1412.3781\n",
      "2018-02-10 00:58:48.214372 Iter 9600: Training Loss = 89931.9141\n",
      "2018-02-10 01:01:36.786442 Iter 9700: Training Loss = 0.4071\n",
      "2018-02-10 01:04:24.847482 Iter 9800: Training Loss = 2251.3167\n",
      "2018-02-10 01:07:13.366187 Iter 9900: Training Loss = 0.4071\n",
      "2018-02-10 01:10:02.404724 Iter 10000: Training Loss = 0.4071\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-10000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-10000\n",
      "2018-02-10 01:12:54.348402 Iter 10100: Training Loss = 15343.2520\n",
      "2018-02-10 01:15:43.268408 Iter 10200: Training Loss = 95444.4141\n",
      "2018-02-10 01:18:32.070355 Iter 10300: Training Loss = 181201.4375\n",
      "2018-02-10 01:21:20.317212 Iter 10400: Training Loss = 144.9502\n",
      "2018-02-10 01:24:09.005755 Iter 10500: Training Loss = 574.4850\n",
      "2018-02-10 01:26:57.484168 Iter 10600: Training Loss = 0.4071\n",
      "2018-02-10 01:29:45.883798 Iter 10700: Training Loss = 47222.1797\n",
      "2018-02-10 01:32:34.149868 Iter 10800: Training Loss = 2.4337\n",
      "2018-02-10 01:35:21.805795 Iter 10900: Training Loss = 141975.1094\n",
      "2018-02-10 01:38:10.508396 Iter 11000: Training Loss = 50693.2891\n",
      "2018-02-10 01:40:59.531094 Iter 11100: Training Loss = 3360.5842\n",
      "2018-02-10 01:43:47.629687 Iter 11200: Training Loss = 6.9216\n",
      "2018-02-10 01:46:35.701491 Iter 11300: Training Loss = 0.4071\n",
      "2018-02-10 01:49:23.852924 Iter 11400: Training Loss = 0.4071\n",
      "2018-02-10 01:52:13.000446 Iter 11500: Training Loss = 306.8936\n",
      "2018-02-10 01:55:00.940042 Iter 11600: Training Loss = 13289.7900\n",
      "2018-02-10 01:57:48.392329 Iter 11700: Training Loss = 64942.8984\n",
      "2018-02-10 02:00:36.281843 Iter 11800: Training Loss = 3660.9260\n",
      "2018-02-10 02:03:24.452092 Iter 11900: Training Loss = 941.2330\n",
      "2018-02-10 02:06:12.516479 Iter 12000: Training Loss = 376.3507\n",
      "2018-02-10 02:09:00.950399 Iter 12100: Training Loss = 2554.2737\n",
      "2018-02-10 02:11:49.177654 Iter 12200: Training Loss = 703.5660\n",
      "2018-02-10 02:14:38.009202 Iter 12300: Training Loss = 0.4071\n",
      "2018-02-10 02:17:26.729815 Iter 12400: Training Loss = 151542.6094\n",
      "2018-02-10 02:20:15.161352 Iter 12500: Training Loss = 24.1726\n",
      "2018-02-10 02:23:03.416224 Iter 12600: Training Loss = 949.1530\n",
      "2018-02-10 02:25:52.691703 Iter 12700: Training Loss = 93840.7656\n",
      "2018-02-10 02:28:40.242335 Iter 12800: Training Loss = 3.9503\n",
      "2018-02-10 02:31:29.146845 Iter 12900: Training Loss = 0.4071\n",
      "2018-02-10 02:34:17.674001 Iter 13000: Training Loss = 14041.5264\n",
      "2018-02-10 02:37:05.729167 Iter 13100: Training Loss = 79753.7188\n",
      "2018-02-10 02:39:52.910870 Iter 13200: Training Loss = 5337.8706\n",
      "2018-02-10 02:42:41.632272 Iter 13300: Training Loss = 31864.9707\n",
      "2018-02-10 02:45:29.260676 Iter 13400: Training Loss = 45052.7930\n",
      "2018-02-10 02:48:16.339446 Iter 13500: Training Loss = 0.4071\n",
      "2018-02-10 02:51:05.110057 Iter 13600: Training Loss = 1028.9269\n",
      "2018-02-10 02:53:52.748385 Iter 13700: Training Loss = 7233.5332\n",
      "2018-02-10 02:56:40.799060 Iter 13800: Training Loss = 1198.8328\n",
      "2018-02-10 02:59:29.196689 Iter 13900: Training Loss = 71986.4219\n",
      "2018-02-10 03:02:17.897971 Iter 14000: Training Loss = 18870.6699\n",
      "2018-02-10 03:05:06.311469 Iter 14100: Training Loss = 587.1754\n",
      "2018-02-10 03:07:53.854457 Iter 14200: Training Loss = 316.0511\n",
      "2018-02-10 03:10:41.994094 Iter 14300: Training Loss = 3908.0215\n",
      "2018-02-10 03:13:29.983882 Iter 14400: Training Loss = 0.4071\n",
      "2018-02-10 03:16:18.530194 Iter 14500: Training Loss = 308.1954\n",
      "2018-02-10 03:19:07.420427 Iter 14600: Training Loss = 0.4071\n",
      "2018-02-10 03:21:55.554070 Iter 14700: Training Loss = 0.4071\n",
      "2018-02-10 03:24:43.977044 Iter 14800: Training Loss = 181.6192\n",
      "2018-02-10 03:27:32.580628 Iter 14900: Training Loss = 48689.9766\n",
      "2018-02-10 03:30:20.852254 Iter 15000: Training Loss = 4026.2322\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-15000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-15000\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network with strong side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 1, learning_rate, logs_path, max_training_iters_1, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, test_image_path=test_image,\n",
    "                           ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = weak/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = weak/conv1/conv1_1/weights:0, shape = (3, 3, 4, 64)\n",
      "   name = weak/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = weak/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = weak/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = weak/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = weak/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = weak/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = weak/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = weak/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = weak/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = weak/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = weak/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = weak/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = weak/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = weak/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = weak/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = weak/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = weak/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = weak/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = weak/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = weak/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = weak/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = weak/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = weak/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = weak/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = weak/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from previous checkpoint...\n",
      "INFO:tensorflow:Restoring parameters from models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-15000\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-02-10 03:33:25.205649 Iter 15100: Training Loss = 0.4071\n",
      "2018-02-10 03:36:13.506143 Iter 15200: Training Loss = 399.9398\n",
      "2018-02-10 03:39:01.017823 Iter 15300: Training Loss = 18514.2266\n",
      "2018-02-10 03:41:48.661755 Iter 15400: Training Loss = 14121.2236\n",
      "2018-02-10 03:44:35.724865 Iter 15500: Training Loss = 3109.1460\n",
      "2018-02-10 03:47:24.242375 Iter 15600: Training Loss = 48463.3125\n",
      "2018-02-10 03:50:12.651089 Iter 15700: Training Loss = 2842.9153\n",
      "2018-02-10 03:53:00.658599 Iter 15800: Training Loss = 21976.9141\n",
      "2018-02-10 03:55:48.107593 Iter 15900: Training Loss = 0.4071\n",
      "2018-02-10 03:58:35.704630 Iter 16000: Training Loss = 0.4071\n",
      "2018-02-10 04:01:23.773069 Iter 16100: Training Loss = 0.4071\n",
      "2018-02-10 04:04:11.454263 Iter 16200: Training Loss = 0.4071\n",
      "2018-02-10 04:06:58.843732 Iter 16300: Training Loss = 0.4071\n",
      "2018-02-10 04:09:47.475800 Iter 16400: Training Loss = 0.4071\n",
      "2018-02-10 04:12:35.963022 Iter 16500: Training Loss = 853.3666\n",
      "2018-02-10 04:15:23.522041 Iter 16600: Training Loss = 429.7440\n",
      "2018-02-10 04:18:11.997614 Iter 16700: Training Loss = 177.6553\n",
      "2018-02-10 04:20:59.943629 Iter 16800: Training Loss = 53949.8750\n",
      "2018-02-10 04:23:48.514254 Iter 16900: Training Loss = 5.0927\n",
      "2018-02-10 04:26:36.699287 Iter 17000: Training Loss = 0.4072\n",
      "2018-02-10 04:29:24.463214 Iter 17100: Training Loss = 2643.6177\n",
      "2018-02-10 04:32:12.619130 Iter 17200: Training Loss = 15499.7588\n",
      "2018-02-10 04:34:59.832634 Iter 17300: Training Loss = 1434.5248\n",
      "2018-02-10 04:37:48.064233 Iter 17400: Training Loss = 0.4072\n",
      "2018-02-10 04:40:36.246722 Iter 17500: Training Loss = 19749.1367\n",
      "2018-02-10 04:43:23.404646 Iter 17600: Training Loss = 16937.6367\n",
      "2018-02-10 04:46:11.170676 Iter 17700: Training Loss = 23483.7598\n",
      "2018-02-10 04:49:01.818287 Iter 17800: Training Loss = 246.5621\n",
      "2018-02-10 04:51:49.701992 Iter 17900: Training Loss = 6094.3408\n",
      "2018-02-10 04:54:37.358837 Iter 18000: Training Loss = 117818.0156\n",
      "2018-02-10 04:57:24.129068 Iter 18100: Training Loss = 2568.0498\n",
      "2018-02-10 05:00:11.504568 Iter 18200: Training Loss = 4635.4609\n",
      "2018-02-10 05:02:59.176718 Iter 18300: Training Loss = 87005.3984\n",
      "2018-02-10 05:05:47.528794 Iter 18400: Training Loss = 7038.6763\n",
      "2018-02-10 05:08:35.824198 Iter 18500: Training Loss = 4155.5811\n",
      "2018-02-10 05:11:24.650759 Iter 18600: Training Loss = 1910.0977\n",
      "2018-02-10 05:14:11.718051 Iter 18700: Training Loss = 3388.2864\n",
      "2018-02-10 05:16:59.795643 Iter 18800: Training Loss = 0.7482\n",
      "2018-02-10 05:19:46.918572 Iter 18900: Training Loss = 15837.5361\n",
      "2018-02-10 05:22:34.242022 Iter 19000: Training Loss = 7025.9785\n",
      "2018-02-10 05:25:22.347212 Iter 19100: Training Loss = 6094.9028\n",
      "2018-02-10 05:28:10.318308 Iter 19200: Training Loss = 20622.0840\n",
      "2018-02-10 05:30:57.246569 Iter 19300: Training Loss = 37576.9062\n",
      "2018-02-10 05:33:45.200291 Iter 19400: Training Loss = 4868.4321\n",
      "2018-02-10 05:36:33.341127 Iter 19500: Training Loss = 66204.5703\n",
      "2018-02-10 05:39:21.292609 Iter 19600: Training Loss = 18617.3711\n",
      "2018-02-10 05:42:09.376557 Iter 19700: Training Loss = 69905.9375\n",
      "2018-02-10 05:44:56.141204 Iter 19800: Training Loss = 16218.4111\n",
      "2018-02-10 05:47:43.906079 Iter 19900: Training Loss = 567.0744\n",
      "2018-02-10 05:50:32.336902 Iter 20000: Training Loss = 157.9598\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-20000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-20000\n",
      "2018-02-10 05:53:23.838379 Iter 20100: Training Loss = 6939.7964\n",
      "2018-02-10 05:56:11.653176 Iter 20200: Training Loss = 129.6137\n",
      "2018-02-10 05:58:58.700716 Iter 20300: Training Loss = 30.3492\n",
      "2018-02-10 06:01:46.463833 Iter 20400: Training Loss = 2012.3745\n",
      "2018-02-10 06:04:33.276609 Iter 20500: Training Loss = 2648.7783\n",
      "2018-02-10 06:07:20.206573 Iter 20600: Training Loss = 0.4073\n",
      "2018-02-10 06:10:08.494642 Iter 20700: Training Loss = 15138.9512\n",
      "2018-02-10 06:12:56.445503 Iter 20800: Training Loss = 2025.9534\n",
      "2018-02-10 06:15:44.058585 Iter 20900: Training Loss = 589.7366\n",
      "2018-02-10 06:18:31.442242 Iter 21000: Training Loss = 1658.9557\n",
      "2018-02-10 06:21:19.449317 Iter 21100: Training Loss = 1647.6316\n",
      "2018-02-10 06:24:07.759951 Iter 21200: Training Loss = 0.4073\n",
      "2018-02-10 06:26:55.155403 Iter 21300: Training Loss = 806.6334\n",
      "2018-02-10 06:29:43.541211 Iter 21400: Training Loss = 0.4073\n",
      "2018-02-10 06:32:34.668916 Iter 21500: Training Loss = 32703.2910\n",
      "2018-02-10 06:35:36.425725 Iter 21600: Training Loss = 0.4073\n",
      "2018-02-10 06:38:37.701080 Iter 21700: Training Loss = 3713.3901\n",
      "2018-02-10 06:42:17.791295 Iter 21800: Training Loss = 57524.9062\n",
      "2018-02-10 06:45:19.786045 Iter 21900: Training Loss = 11433.1260\n",
      "2018-02-10 06:48:20.845034 Iter 22000: Training Loss = 1001.6031\n",
      "2018-02-10 06:51:20.723678 Iter 22100: Training Loss = 419.3820\n",
      "2018-02-10 06:54:21.682218 Iter 22200: Training Loss = 1224.1101\n",
      "2018-02-10 06:57:22.538324 Iter 22300: Training Loss = 7330.3145\n",
      "2018-02-10 07:00:23.463916 Iter 22400: Training Loss = 5219.2002\n",
      "2018-02-10 07:03:23.883383 Iter 22500: Training Loss = 8668.7354\n",
      "2018-02-10 07:06:24.870079 Iter 22600: Training Loss = 19217.5273\n",
      "2018-02-10 07:09:25.155027 Iter 22700: Training Loss = 9388.4453\n",
      "2018-02-10 07:12:26.185062 Iter 22800: Training Loss = 0.4073\n",
      "2018-02-10 07:15:27.024161 Iter 22900: Training Loss = 0.4073\n",
      "2018-02-10 07:18:27.411163 Iter 23000: Training Loss = 4998.5493\n",
      "2018-02-10 07:21:28.775064 Iter 23100: Training Loss = 6679.8877\n",
      "2018-02-10 07:24:30.108624 Iter 23200: Training Loss = 18.2480\n",
      "2018-02-10 07:27:31.579645 Iter 23300: Training Loss = 12233.7959\n",
      "2018-02-10 07:30:33.064941 Iter 23400: Training Loss = 19417.2090\n",
      "2018-02-10 07:33:34.552617 Iter 23500: Training Loss = 4535.4512\n",
      "2018-02-10 07:36:36.323041 Iter 23600: Training Loss = 66253.7812\n",
      "2018-02-10 07:39:38.122036 Iter 23700: Training Loss = 8132.3110\n",
      "2018-02-10 07:42:38.084262 Iter 23800: Training Loss = 52206.5781\n",
      "2018-02-10 07:45:39.116938 Iter 23900: Training Loss = 3517.7739\n",
      "2018-02-10 07:48:40.236997 Iter 24000: Training Loss = 23871.8223\n",
      "2018-02-10 07:51:42.029388 Iter 24100: Training Loss = 14848.8008\n",
      "2018-02-10 07:54:42.990377 Iter 24200: Training Loss = 16288.7549\n",
      "2018-02-10 07:57:43.736350 Iter 24300: Training Loss = 74263.0469\n",
      "2018-02-10 08:00:44.829698 Iter 24400: Training Loss = 0.4074\n",
      "2018-02-10 08:03:45.565029 Iter 24500: Training Loss = 34359.9922\n",
      "2018-02-10 08:06:46.536142 Iter 24600: Training Loss = 1308.2046\n",
      "2018-02-10 08:09:47.826060 Iter 24700: Training Loss = 0.4074\n",
      "2018-02-10 08:12:49.710342 Iter 24800: Training Loss = 1251.3407\n",
      "2018-02-10 08:15:50.592100 Iter 24900: Training Loss = 221.6953\n",
      "2018-02-10 08:18:52.403376 Iter 25000: Training Loss = 253.1893\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-25000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-25000\n",
      "2018-02-10 08:21:58.117979 Iter 25100: Training Loss = 10526.8682\n",
      "2018-02-10 08:24:58.139731 Iter 25200: Training Loss = 111990.1250\n",
      "2018-02-10 08:27:58.544030 Iter 25300: Training Loss = 1973.2704\n",
      "2018-02-10 08:30:58.945763 Iter 25400: Training Loss = 8335.7783\n",
      "2018-02-10 08:33:59.466459 Iter 25500: Training Loss = 10324.4648\n",
      "2018-02-10 08:37:00.452331 Iter 25600: Training Loss = 0.4074\n",
      "2018-02-10 08:40:01.021657 Iter 25700: Training Loss = 3670.0481\n",
      "2018-02-10 08:43:01.374248 Iter 25800: Training Loss = 1576.5619\n",
      "2018-02-10 08:46:02.805785 Iter 25900: Training Loss = 19535.3457\n",
      "2018-02-10 08:49:04.416688 Iter 26000: Training Loss = 6587.9990\n",
      "2018-02-10 08:52:03.773417 Iter 26100: Training Loss = 15106.6514\n",
      "2018-02-10 08:54:51.182392 Iter 26200: Training Loss = 5555.5977\n",
      "2018-02-10 08:57:38.533048 Iter 26300: Training Loss = 1741.1052\n",
      "2018-02-10 09:00:26.015159 Iter 26400: Training Loss = 0.4074\n",
      "2018-02-10 09:03:14.052282 Iter 26500: Training Loss = 230.2634\n",
      "2018-02-10 09:06:01.890108 Iter 26600: Training Loss = 22513.0957\n",
      "2018-02-10 09:08:49.394536 Iter 26700: Training Loss = 5.5342\n",
      "2018-02-10 09:11:36.485800 Iter 26800: Training Loss = 1723.5270\n",
      "2018-02-10 09:14:24.409744 Iter 26900: Training Loss = 1446.1935\n",
      "2018-02-10 09:17:11.521458 Iter 27000: Training Loss = 44.4143\n",
      "2018-02-10 09:19:59.220251 Iter 27100: Training Loss = 0.4074\n",
      "2018-02-10 09:22:46.278206 Iter 27200: Training Loss = 1081.3938\n",
      "2018-02-10 09:25:34.041005 Iter 27300: Training Loss = 22016.9453\n",
      "2018-02-10 09:28:21.392657 Iter 27400: Training Loss = 13304.7207\n",
      "2018-02-10 09:31:08.844102 Iter 27500: Training Loss = 9633.4121\n",
      "2018-02-10 09:33:56.388681 Iter 27600: Training Loss = 1010.7098\n",
      "2018-02-10 09:36:43.541937 Iter 27700: Training Loss = 26233.6992\n",
      "2018-02-10 09:39:30.775915 Iter 27800: Training Loss = 2435.7659\n",
      "2018-02-10 09:42:18.928465 Iter 27900: Training Loss = 12191.2510\n",
      "2018-02-10 09:45:05.714609 Iter 28000: Training Loss = 20627.4980\n",
      "2018-02-10 09:47:52.231726 Iter 28100: Training Loss = 62137.9531\n",
      "2018-02-10 09:50:40.283731 Iter 28200: Training Loss = 20568.9043\n",
      "2018-02-10 09:53:27.129939 Iter 28300: Training Loss = 13918.0654\n",
      "2018-02-10 09:56:13.812782 Iter 28400: Training Loss = 9196.0771\n",
      "2018-02-10 09:59:01.745283 Iter 28500: Training Loss = 1.8419\n",
      "2018-02-10 10:01:49.762149 Iter 28600: Training Loss = 6601.4272\n",
      "2018-02-10 10:04:36.237098 Iter 28700: Training Loss = 40236.4062\n",
      "2018-02-10 10:07:24.120510 Iter 28800: Training Loss = 2924.2058\n",
      "2018-02-10 10:10:11.345553 Iter 28900: Training Loss = 0.4074\n",
      "2018-02-10 10:12:58.218287 Iter 29000: Training Loss = 0.4074\n",
      "2018-02-10 10:15:45.659439 Iter 29100: Training Loss = 0.4074\n",
      "2018-02-10 10:18:33.569686 Iter 29200: Training Loss = 28505.3730\n",
      "2018-02-10 10:21:20.996773 Iter 29300: Training Loss = 661.9738\n",
      "2018-02-10 10:24:08.075511 Iter 29400: Training Loss = 0.4074\n",
      "2018-02-10 10:26:55.433181 Iter 29500: Training Loss = 47168.1875\n",
      "2018-02-10 10:29:42.211724 Iter 29600: Training Loss = 29948.8613\n",
      "2018-02-10 10:32:29.464080 Iter 29700: Training Loss = 45229.5859\n",
      "2018-02-10 10:35:16.594186 Iter 29800: Training Loss = 2027.2356\n",
      "2018-02-10 10:38:03.832392 Iter 29900: Training Loss = 1352.9026\n",
      "2018-02-10 10:40:51.864669 Iter 30000: Training Loss = 17593.6855\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-30000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-30000\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network with weak side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(max_training_iters_1, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 2, learning_rate, logs_path, max_training_iters_2, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, resume_training=True,\n",
    "                           test_image_path=test_image, ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Layers:\n",
      "   name = weak/conv1/conv1_1/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv1/conv1_2/Relu:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/pool1/MaxPool:0, shape = (1, ?, ?, 64)\n",
      "   name = weak/conv2/conv2_1/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv2/conv2_2/Relu:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/pool2/MaxPool:0, shape = (1, ?, ?, 128)\n",
      "   name = weak/conv3/conv3_1/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_2/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv3/conv3_3/Relu:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/pool3/MaxPool:0, shape = (1, ?, ?, 256)\n",
      "   name = weak/conv4/conv4_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv4/conv4_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/pool4/MaxPool:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_1/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_2/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv5/conv5_3/Relu:0, shape = (1, ?, ?, 512)\n",
      "   name = weak/conv2_2_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv3_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv4_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/conv5_3_16/BiasAdd:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-dsn_2/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_2-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_3-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_1:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_4-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_2:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-dsn_5-up/conv2d_transpose:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/Reshape_3:0, shape = (1, ?, ?, 1)\n",
      "   name = weak/score-multi2-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_4:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi3-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_5:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi4-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_6:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/score-multi5-up/conv2d_transpose:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/Reshape_7:0, shape = (1, ?, ?, 16)\n",
      "   name = weak/upscore-fuse/BiasAdd:0, shape = (1, ?, ?, 1)\n",
      "Network Parameters:\n",
      "   name = weak/conv1/conv1_1/weights:0, shape = (3, 3, 4, 64)\n",
      "   name = weak/conv1/conv1_1/biases:0, shape = (64,)\n",
      "   name = weak/conv1/conv1_2/weights:0, shape = (3, 3, 64, 64)\n",
      "   name = weak/conv1/conv1_2/biases:0, shape = (64,)\n",
      "   name = weak/conv2/conv2_1/weights:0, shape = (3, 3, 64, 128)\n",
      "   name = weak/conv2/conv2_1/biases:0, shape = (128,)\n",
      "   name = weak/conv2/conv2_2/weights:0, shape = (3, 3, 128, 128)\n",
      "   name = weak/conv2/conv2_2/biases:0, shape = (128,)\n",
      "   name = weak/conv3/conv3_1/weights:0, shape = (3, 3, 128, 256)\n",
      "   name = weak/conv3/conv3_1/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_2/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_2/biases:0, shape = (256,)\n",
      "   name = weak/conv3/conv3_3/weights:0, shape = (3, 3, 256, 256)\n",
      "   name = weak/conv3/conv3_3/biases:0, shape = (256,)\n",
      "   name = weak/conv4/conv4_1/weights:0, shape = (3, 3, 256, 512)\n",
      "   name = weak/conv4/conv4_1/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_2/biases:0, shape = (512,)\n",
      "   name = weak/conv4/conv4_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv4/conv4_3/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_1/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_1/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_2/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_2/biases:0, shape = (512,)\n",
      "   name = weak/conv5/conv5_3/weights:0, shape = (3, 3, 512, 512)\n",
      "   name = weak/conv5/conv5_3/biases:0, shape = (512,)\n",
      "   name = weak/conv2_2_16/weights:0, shape = (3, 3, 128, 16)\n",
      "   name = weak/conv2_2_16/biases:0, shape = (16,)\n",
      "   name = weak/conv3_3_16/weights:0, shape = (3, 3, 256, 16)\n",
      "   name = weak/conv3_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv4_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv4_3_16/biases:0, shape = (16,)\n",
      "   name = weak/conv5_3_16/weights:0, shape = (3, 3, 512, 16)\n",
      "   name = weak/conv5_3_16/biases:0, shape = (16,)\n",
      "   name = weak/score-dsn_2/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_2/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_3/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_3/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_4/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_4/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_5/weights:0, shape = (1, 1, 16, 1)\n",
      "   name = weak/score-dsn_5/biases:0, shape = (1,)\n",
      "   name = weak/score-dsn_2-up/weights:0, shape = (4, 4, 1, 1)\n",
      "   name = weak/score-dsn_3-up/weights:0, shape = (8, 8, 1, 1)\n",
      "   name = weak/score-dsn_4-up/weights:0, shape = (16, 16, 1, 1)\n",
      "   name = weak/score-dsn_5-up/weights:0, shape = (32, 32, 1, 1)\n",
      "   name = weak/score-multi2-up/weights:0, shape = (4, 4, 16, 16)\n",
      "   name = weak/score-multi3-up/weights:0, shape = (8, 8, 16, 16)\n",
      "   name = weak/score-multi4-up/weights:0, shape = (16, 16, 16, 16)\n",
      "   name = weak/score-multi5-up/weights:0, shape = (32, 32, 16, 16)\n",
      "   name = weak/upscore-fuse/weights:0, shape = (1, 1, 64, 1)\n",
      "   name = weak/upscore-fuse/biases:0, shape = (1,)\n",
      "Init variable\n",
      "Initializing from previous checkpoint...\n",
      "INFO:tensorflow:Restoring parameters from models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-30000\n",
      "Weights initialized\n",
      "Start training\n",
      "2018-02-10 10:43:37.257429 Iter 30100: Training Loss = 9154.3584\n",
      "2018-02-10 10:46:09.995806 Iter 30200: Training Loss = 9386.7158\n",
      "2018-02-10 10:48:41.062202 Iter 30300: Training Loss = 112.2684\n",
      "2018-02-10 10:51:12.294751 Iter 30400: Training Loss = 354.9545\n",
      "2018-02-10 10:53:43.343451 Iter 30500: Training Loss = 100.1299\n",
      "2018-02-10 10:56:14.554769 Iter 30600: Training Loss = 11298.8584\n",
      "2018-02-10 10:58:45.295356 Iter 30700: Training Loss = 8353.5723\n",
      "2018-02-10 11:01:16.514868 Iter 30800: Training Loss = 88.8171\n",
      "2018-02-10 11:03:48.990100 Iter 30900: Training Loss = 0.4074\n",
      "2018-02-10 11:06:20.987729 Iter 31000: Training Loss = 6877.9385\n",
      "2018-02-10 11:08:52.827668 Iter 31100: Training Loss = 25915.0410\n",
      "2018-02-10 11:11:23.478067 Iter 31200: Training Loss = 4946.4170\n",
      "2018-02-10 11:13:55.050098 Iter 31300: Training Loss = 629.2032\n",
      "2018-02-10 11:16:26.731060 Iter 31400: Training Loss = 1142.0702\n",
      "2018-02-10 11:18:57.601377 Iter 31500: Training Loss = 184.0704\n",
      "2018-02-10 11:21:29.675491 Iter 31600: Training Loss = 14461.5518\n",
      "2018-02-10 11:24:00.822923 Iter 31700: Training Loss = 0.4075\n",
      "2018-02-10 11:26:32.608668 Iter 31800: Training Loss = 106.7872\n",
      "2018-02-10 11:29:03.947761 Iter 31900: Training Loss = 0.4075\n",
      "2018-02-10 11:31:34.986778 Iter 32000: Training Loss = 208.0148\n",
      "2018-02-10 11:34:05.642913 Iter 32100: Training Loss = 3.1802\n",
      "2018-02-10 11:36:37.839500 Iter 32200: Training Loss = 4644.1846\n",
      "2018-02-10 11:39:09.517476 Iter 32300: Training Loss = 0.4075\n",
      "2018-02-10 11:41:41.778193 Iter 32400: Training Loss = 3291.2507\n",
      "2018-02-10 11:44:12.841911 Iter 32500: Training Loss = 1981.6377\n",
      "2018-02-10 11:46:44.564686 Iter 32600: Training Loss = 19426.2383\n",
      "2018-02-10 11:49:15.055156 Iter 32700: Training Loss = 5056.5615\n",
      "2018-02-10 11:51:47.541739 Iter 32800: Training Loss = 0.6240\n",
      "2018-02-10 11:54:19.457548 Iter 32900: Training Loss = 616.2773\n",
      "2018-02-10 11:56:51.624303 Iter 33000: Training Loss = 119.6234\n",
      "2018-02-10 11:59:22.602872 Iter 33100: Training Loss = 63.2594\n",
      "2018-02-10 12:01:53.718438 Iter 33200: Training Loss = 0.4075\n",
      "2018-02-10 12:04:25.063367 Iter 33300: Training Loss = 8433.1582\n",
      "2018-02-10 12:06:57.303302 Iter 33400: Training Loss = 6461.9487\n",
      "2018-02-10 12:09:28.810922 Iter 33500: Training Loss = 0.5113\n",
      "2018-02-10 12:12:00.797190 Iter 33600: Training Loss = 826.0049\n",
      "2018-02-10 12:14:32.490603 Iter 33700: Training Loss = 751.9801\n",
      "2018-02-10 12:17:04.400524 Iter 33800: Training Loss = 548.2106\n",
      "2018-02-10 12:19:35.264881 Iter 33900: Training Loss = 1872.2078\n",
      "2018-02-10 12:22:06.842141 Iter 34000: Training Loss = 8051.1313\n",
      "2018-02-10 12:24:38.002860 Iter 34100: Training Loss = 2098.1526\n",
      "2018-02-10 12:27:09.041730 Iter 34200: Training Loss = 1.9504\n",
      "2018-02-10 12:29:40.233734 Iter 34300: Training Loss = 0.4075\n",
      "2018-02-10 12:32:12.427990 Iter 34400: Training Loss = 0.4075\n",
      "2018-02-10 12:34:44.172063 Iter 34500: Training Loss = 0.4075\n",
      "2018-02-10 12:37:16.156525 Iter 34600: Training Loss = 31718.4922\n",
      "2018-02-10 12:39:47.353167 Iter 34700: Training Loss = 0.4075\n",
      "2018-02-10 12:42:17.961878 Iter 34800: Training Loss = 2572.1372\n",
      "2018-02-10 12:44:48.935991 Iter 34900: Training Loss = 5393.4233\n",
      "2018-02-10 12:47:21.619111 Iter 35000: Training Loss = 0.4076\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-35000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-35000\n",
      "2018-02-10 12:49:55.714719 Iter 35100: Training Loss = 27005.1914\n",
      "2018-02-10 12:52:26.917791 Iter 35200: Training Loss = 794.7064\n",
      "2018-02-10 12:54:58.087698 Iter 35300: Training Loss = 123.1343\n",
      "2018-02-10 12:57:30.136212 Iter 35400: Training Loss = 0.4076\n",
      "2018-02-10 13:00:01.303330 Iter 35500: Training Loss = 4999.6416\n",
      "2018-02-10 13:02:32.810201 Iter 35600: Training Loss = 2818.4783\n",
      "2018-02-10 13:05:04.169022 Iter 35700: Training Loss = 4126.8970\n",
      "2018-02-10 13:07:35.493223 Iter 35800: Training Loss = 9569.2256\n",
      "2018-02-10 13:10:07.195797 Iter 35900: Training Loss = 0.4076\n",
      "2018-02-10 13:12:43.434638 Iter 36000: Training Loss = 14674.7471\n",
      "2018-02-10 13:15:28.032615 Iter 36100: Training Loss = 21.4467\n",
      "2018-02-10 13:18:11.886160 Iter 36200: Training Loss = 11769.4883\n",
      "2018-02-10 13:20:55.957733 Iter 36300: Training Loss = 167.1800\n",
      "2018-02-10 13:23:40.511074 Iter 36400: Training Loss = 7357.2808\n",
      "2018-02-10 13:26:24.554744 Iter 36500: Training Loss = 0.4076\n",
      "2018-02-10 13:29:09.027846 Iter 36600: Training Loss = 1979.9413\n",
      "2018-02-10 13:31:53.240746 Iter 36700: Training Loss = 5430.9619\n",
      "2018-02-10 13:34:37.032098 Iter 36800: Training Loss = 0.4076\n",
      "2018-02-10 13:37:21.084176 Iter 36900: Training Loss = 19.3613\n",
      "2018-02-10 13:40:05.530345 Iter 37000: Training Loss = 0.4076\n",
      "2018-02-10 13:42:49.937305 Iter 37100: Training Loss = 3835.1304\n",
      "2018-02-10 13:45:34.100385 Iter 37200: Training Loss = 1082.7992\n",
      "2018-02-10 13:48:18.171324 Iter 37300: Training Loss = 2542.4890\n",
      "2018-02-10 13:51:02.130733 Iter 37400: Training Loss = 0.4076\n",
      "2018-02-10 13:53:46.236775 Iter 37500: Training Loss = 7199.7637\n",
      "2018-02-10 13:56:30.713078 Iter 37600: Training Loss = 0.4076\n",
      "2018-02-10 13:59:16.035582 Iter 37700: Training Loss = 0.4076\n",
      "2018-02-10 14:02:00.461809 Iter 37800: Training Loss = 22505.6309\n",
      "2018-02-10 14:04:44.504100 Iter 37900: Training Loss = 5148.9751\n",
      "2018-02-10 14:07:28.286757 Iter 38000: Training Loss = 33.2798\n",
      "2018-02-10 14:10:13.177022 Iter 38100: Training Loss = 3471.8882\n",
      "2018-02-10 14:12:58.037744 Iter 38200: Training Loss = 1006.0919\n",
      "2018-02-10 14:15:42.347817 Iter 38300: Training Loss = 226.2232\n",
      "2018-02-10 14:18:26.691254 Iter 38400: Training Loss = 46367.2578\n",
      "2018-02-10 14:21:10.621745 Iter 38500: Training Loss = 10332.6309\n",
      "2018-02-10 14:23:55.724609 Iter 38600: Training Loss = 2160.1252\n",
      "2018-02-10 14:26:39.698490 Iter 38700: Training Loss = 598.4289\n",
      "2018-02-10 14:29:23.390684 Iter 38800: Training Loss = 445.7434\n",
      "2018-02-10 14:32:07.705689 Iter 38900: Training Loss = 51.3324\n",
      "2018-02-10 14:34:51.749806 Iter 39000: Training Loss = 515.0573\n",
      "2018-02-10 14:37:35.724727 Iter 39100: Training Loss = 5572.9800\n",
      "2018-02-10 14:40:19.960730 Iter 39200: Training Loss = 12436.3623\n",
      "2018-02-10 14:43:03.367998 Iter 39300: Training Loss = 7096.1089\n",
      "2018-02-10 14:45:48.547729 Iter 39400: Training Loss = 9358.7676\n",
      "2018-02-10 14:48:32.481794 Iter 39500: Training Loss = 6897.9014\n",
      "2018-02-10 14:51:16.689242 Iter 39600: Training Loss = 405.6731\n",
      "2018-02-10 14:54:01.191887 Iter 39700: Training Loss = 263.4168\n",
      "2018-02-10 14:56:45.205045 Iter 39800: Training Loss = 379.8699\n",
      "2018-02-10 14:59:30.359132 Iter 39900: Training Loss = 3640.9783\n",
      "2018-02-10 15:02:15.160809 Iter 40000: Training Loss = 972.5306\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-40000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-40000\n",
      "2018-02-10 15:05:02.293583 Iter 40100: Training Loss = 1295.4769\n",
      "2018-02-10 15:07:45.340566 Iter 40200: Training Loss = 17653.8887\n",
      "2018-02-10 15:10:29.552732 Iter 40300: Training Loss = 0.4077\n",
      "2018-02-10 15:13:13.619804 Iter 40400: Training Loss = 0.4077\n",
      "2018-02-10 15:15:57.465831 Iter 40500: Training Loss = 1472.0503\n",
      "2018-02-10 15:18:41.266011 Iter 40600: Training Loss = 0.4077\n",
      "2018-02-10 15:21:25.771210 Iter 40700: Training Loss = 42.3757\n",
      "2018-02-10 15:24:09.738048 Iter 40800: Training Loss = 238.1475\n",
      "2018-02-10 15:26:54.541498 Iter 40900: Training Loss = 6.9685\n",
      "2018-02-10 15:29:38.050899 Iter 41000: Training Loss = 707.3343\n",
      "2018-02-10 15:32:22.322819 Iter 41100: Training Loss = 987.4252\n",
      "2018-02-10 15:35:06.381055 Iter 41200: Training Loss = 301.8680\n",
      "2018-02-10 15:37:49.587701 Iter 41300: Training Loss = 6214.3887\n",
      "2018-02-10 15:40:33.359600 Iter 41400: Training Loss = 0.4077\n",
      "2018-02-10 15:43:16.033644 Iter 41500: Training Loss = 0.4077\n",
      "2018-02-10 15:45:59.867361 Iter 41600: Training Loss = 372.2342\n",
      "2018-02-10 15:48:43.930921 Iter 41700: Training Loss = 0.6017\n",
      "2018-02-10 15:51:28.000726 Iter 41800: Training Loss = 1812.9663\n",
      "2018-02-10 15:54:12.375186 Iter 41900: Training Loss = 2858.1467\n",
      "2018-02-10 15:56:56.018254 Iter 42000: Training Loss = 575.3512\n",
      "2018-02-10 15:59:40.063794 Iter 42100: Training Loss = 17091.6348\n",
      "2018-02-10 16:02:23.782763 Iter 42200: Training Loss = 0.9747\n",
      "2018-02-10 16:05:06.831602 Iter 42300: Training Loss = 6163.8999\n",
      "2018-02-10 16:07:50.506048 Iter 42400: Training Loss = 0.4077\n",
      "2018-02-10 16:10:34.172825 Iter 42500: Training Loss = 27.5166\n",
      "2018-02-10 16:13:17.959834 Iter 42600: Training Loss = 14258.3438\n",
      "2018-02-10 16:16:01.268859 Iter 42700: Training Loss = 1152.7903\n",
      "2018-02-10 16:18:44.992758 Iter 42800: Training Loss = 413.4209\n",
      "2018-02-10 16:21:29.352892 Iter 42900: Training Loss = 122.8403\n",
      "2018-02-10 16:24:13.658249 Iter 43000: Training Loss = 1103.9174\n",
      "2018-02-10 16:26:56.852026 Iter 43100: Training Loss = 1165.8955\n",
      "2018-02-10 16:29:40.924686 Iter 43200: Training Loss = 7549.2451\n",
      "2018-02-10 16:32:24.426733 Iter 43300: Training Loss = 12878.2861\n",
      "2018-02-10 16:35:08.066923 Iter 43400: Training Loss = 102.2604\n",
      "2018-02-10 16:37:52.317218 Iter 43500: Training Loss = 0.4077\n",
      "2018-02-10 16:40:35.632781 Iter 43600: Training Loss = 5217.3457\n",
      "2018-02-10 16:43:19.069394 Iter 43700: Training Loss = 5237.6860\n",
      "2018-02-10 16:46:03.021159 Iter 43800: Training Loss = 2726.6953\n",
      "2018-02-10 16:48:46.762469 Iter 43900: Training Loss = 23251.9941\n",
      "2018-02-10 16:51:31.622238 Iter 44000: Training Loss = 747.0646\n",
      "2018-02-10 16:54:15.370649 Iter 44100: Training Loss = 3731.4492\n",
      "2018-02-10 16:56:59.616775 Iter 44200: Training Loss = 145.2042\n",
      "2018-02-10 16:59:43.794775 Iter 44300: Training Loss = 0.4077\n",
      "2018-02-10 17:02:27.555976 Iter 44400: Training Loss = 1006.6595\n",
      "2018-02-10 17:05:11.699470 Iter 44500: Training Loss = 0.4077\n",
      "2018-02-10 17:07:55.344934 Iter 44600: Training Loss = 15982.8584\n",
      "2018-02-10 17:10:39.690606 Iter 44700: Training Loss = 3609.2595\n",
      "2018-02-10 17:13:23.778295 Iter 44800: Training Loss = 4744.0972\n",
      "2018-02-10 17:16:07.138267 Iter 44900: Training Loss = 9.8419\n",
      "2018-02-10 17:18:51.671490 Iter 45000: Training Loss = 4303.0024\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-45000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-45000\n",
      "2018-02-10 17:21:38.808088 Iter 45100: Training Loss = 3.7233\n",
      "2018-02-10 17:24:22.411885 Iter 45200: Training Loss = 692.3495\n",
      "2018-02-10 17:27:06.432757 Iter 45300: Training Loss = 0.4077\n",
      "2018-02-10 17:29:50.547469 Iter 45400: Training Loss = 491.5002\n",
      "2018-02-10 17:32:35.332936 Iter 45500: Training Loss = 12823.0391\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-02-10 17:35:19.317685 Iter 45600: Training Loss = 7019.5088\n",
      "2018-02-10 17:38:03.406183 Iter 45700: Training Loss = 2725.4863\n",
      "2018-02-10 17:40:47.390936 Iter 45800: Training Loss = 1581.2068\n",
      "2018-02-10 17:43:32.264228 Iter 45900: Training Loss = 140.3235\n",
      "2018-02-10 17:46:16.494487 Iter 46000: Training Loss = 1638.9476\n",
      "2018-02-10 17:49:00.334807 Iter 46100: Training Loss = 1850.7594\n",
      "2018-02-10 17:51:44.084685 Iter 46200: Training Loss = 0.4077\n",
      "2018-02-10 17:54:28.920501 Iter 46300: Training Loss = 1457.1787\n",
      "2018-02-10 17:57:12.672795 Iter 46400: Training Loss = 1645.0813\n",
      "2018-02-10 17:59:57.259274 Iter 46500: Training Loss = 0.4077\n",
      "2018-02-10 18:02:41.196020 Iter 46600: Training Loss = 85.9722\n",
      "2018-02-10 18:05:25.667725 Iter 46700: Training Loss = 37406.7969\n",
      "2018-02-10 18:08:01.146626 Iter 46800: Training Loss = 0.4077\n",
      "2018-02-10 18:10:32.594511 Iter 46900: Training Loss = 0.4077\n",
      "2018-02-10 18:13:04.378672 Iter 47000: Training Loss = 49.4253\n",
      "2018-02-10 18:15:35.880831 Iter 47100: Training Loss = 15788.5332\n",
      "2018-02-10 18:18:07.119575 Iter 47200: Training Loss = 0.4077\n",
      "2018-02-10 18:20:38.048060 Iter 47300: Training Loss = 12261.1699\n",
      "2018-02-10 18:23:09.394093 Iter 47400: Training Loss = 0.4077\n",
      "2018-02-10 18:25:40.406360 Iter 47500: Training Loss = 0.4077\n",
      "2018-02-10 18:28:11.393219 Iter 47600: Training Loss = 81.1423\n",
      "2018-02-10 18:30:42.340165 Iter 47700: Training Loss = 20.7484\n",
      "2018-02-10 18:33:13.720680 Iter 47800: Training Loss = 1476.1980\n",
      "2018-02-10 18:35:45.406235 Iter 47900: Training Loss = 0.4077\n",
      "2018-02-10 18:38:16.380841 Iter 48000: Training Loss = 16463.9453\n",
      "2018-02-10 18:40:47.140169 Iter 48100: Training Loss = 306.9000\n",
      "2018-02-10 18:43:18.341008 Iter 48200: Training Loss = 4174.2134\n",
      "2018-02-10 18:45:49.770893 Iter 48300: Training Loss = 86.0774\n",
      "2018-02-10 18:48:21.535607 Iter 48400: Training Loss = 0.4077\n",
      "2018-02-10 18:50:52.298826 Iter 48500: Training Loss = 0.4077\n",
      "2018-02-10 18:53:24.026974 Iter 48600: Training Loss = 174.5946\n",
      "2018-02-10 18:55:55.891317 Iter 48700: Training Loss = 0.4077\n",
      "2018-02-10 18:58:27.026345 Iter 48800: Training Loss = 0.4077\n",
      "2018-02-10 19:00:58.061287 Iter 48900: Training Loss = 22.9389\n",
      "2018-02-10 19:03:28.863348 Iter 49000: Training Loss = 14512.6895\n",
      "2018-02-10 19:06:00.147754 Iter 49100: Training Loss = 0.4077\n",
      "2018-02-10 19:08:31.047258 Iter 49200: Training Loss = 1298.4771\n",
      "2018-02-10 19:11:01.800319 Iter 49300: Training Loss = 184.4155\n",
      "2018-02-10 19:13:33.094442 Iter 49400: Training Loss = 0.4077\n",
      "2018-02-10 19:16:04.596080 Iter 49500: Training Loss = 667.9379\n",
      "2018-02-10 19:18:36.035529 Iter 49600: Training Loss = 3507.9353\n",
      "2018-02-10 19:21:07.180374 Iter 49700: Training Loss = 820.1328\n",
      "2018-02-10 19:23:38.334459 Iter 49800: Training Loss = 476.8495\n",
      "2018-02-10 19:26:09.512505 Iter 49900: Training Loss = 10004.2891\n",
      "2018-02-10 19:28:41.211119 Iter 50000: Training Loss = 219.8448\n",
      "INFO:tensorflow:models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-50000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "Model saved in file: models/vgg_16_4chan_weak\\vgg_16_4chan_weak.ckpt-50000\n",
      "Finished training.\n"
     ]
    }
   ],
   "source": [
    "# Train the network without side outputs supervision\n",
    "with tf.Graph().as_default():\n",
    "    with tf.device('/gpu:' + str(gpu_id)):\n",
    "        global_step = tf.Variable(max_training_iters_2, name='global_step', trainable=False)\n",
    "        learning_rate = tf.train.piecewise_constant(global_step, boundaries, values)\n",
    "        model.train_parent(dataset, imagenet_ckpt, 3, learning_rate, logs_path, max_training_iters_3, save_step,\n",
    "                           display_step, global_step, segnet_stream, iter_mean_grad=iter_mean_grad, resume_training=True,\n",
    "                           test_image_path=test_image, ckpt_name=ckpt_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training losses & learning rate\n",
    "You should see training curves similar to the following:\n",
    "![](img/vgg_16_4chan_weak_dsn2_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_dsn3_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_dsn4_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_dsn5_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_main_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_total_loss.png)\n",
    "\n",
    "![](img/vgg_16_4chan_weak_learning_rate.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}