Error while calculating val_loss using validation_data

[pinakinathc]

I have created new JSON file according to my requirement :

• training.json
• test.json

the model trains using training.json but gives error while calculating val_loss using test.json
I have interchanged training.json with test.json (Reason: I though that I might have made some error while creating test.json)

But when test.json is used as training set and training.json is used as validation set
the model trains smoothly and fails to calculate the val_loss (same as before)

Hence the format of training.json and test.json is not an issue.

Below is the output of the error that I get.

16/17 [===========================>..] - ETA: 1s - loss: 167.9720Traceback (most recent call last):
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
    return fn(*args)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
    status, run_metadata)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape [?,?] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
	 [[Node: time_distributed_8/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@time_distributed_8/TensorArray"], dtype=DT_FLOAT, element_shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"](time_distributed_8/TensorArray, time_distributed_8/TensorArrayStack/range, time_distributed_8/while/Exit_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "trainer.py", line 45, in <module>
    validation_data=validation_data
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/training.py", line 1426, in fit_generator
    initial_epoch=initial_epoch)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/training_generator.py", line 211, in fit_generator
    max_queue_size=max_queue_size)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/training.py", line 1480, in evaluate_generator
    verbose=verbose)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/training_generator.py", line 325, in evaluate_generator
    outs = model.test_on_batch(x, y, sample_weight=sample_weight)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/training.py", line 1261, in test_on_batch
    outputs = self.test_function(ins)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2667, in __call__
    return self._legacy_call(inputs)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2649, in _legacy_call
    **self.session_kwargs)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1140, in _run
    feed_dict_tensor, options, run_metadata)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    run_metadata)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape [?,?] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
	 [[Node: time_distributed_8/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@time_distributed_8/TensorArray"], dtype=DT_FLOAT, element_shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"](time_distributed_8/TensorArray, time_distributed_8/TensorArrayStack/range, time_distributed_8/while/Exit_1)]]

Caused by op 'time_distributed_8/TensorArrayStack/TensorArrayGatherV3', defined at:
  File "trainer.py", line 34, in <module>
    'adv', 'bd', 'od'])
  File "/user1/project/cvpr/pinaki_t/keras/keras-rcnn/keras_rcnn/models/_rcnn.py", line 269, in __init__
    )(output_features)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/engine/base_layer.py", line 460, in __call__
    output = self.call(inputs, **kwargs)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/layers/wrappers.py", line 229, in call
    unroll=False)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2962, in rnn
    outputs = output_ta.stack()
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 893, in stack
    return self._implementation.stack(name=name)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 291, in stack
    return self.gather(math_ops.range(0, self.size()), name=name)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 305, in gather
    element_shape=element_shape)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6011, in tensor_array_gather_v3
    flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
    op_def=op_def)
  File "/user1/project/cvpr/pinaki_t/keras/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

UnimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,?] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
	 [[Node: time_distributed_8/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@time_distributed_8/TensorArray"], dtype=DT_FLOAT, element_shape=[?,?], _device="/job:localhost/replica:0/task:0/device:CPU:0"](time_distributed_8/TensorArray, time_distributed_8/TensorArrayStack/range, time_distributed_8/while/Exit_1)]]

[pinakinathc]

@0x00b1 can you please look into this for a second?