ValueError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

[thoongee]

Hi, I'm trying to test validate_on_lfw following Wiki.
I'm done align the LFW dataset, and I have a hard time on number 6. Run the test.

My computer is Ubuntu Linux.
How can I run the test correctly?

2022-07-11 18:15:04.304549: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
2022-07-11 18:15:04.304574: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: ubuntu
2022-07-11 18:15:04.304579: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: ubuntu
2022-07-11 18:15:04.304641: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 510.47.3
2022-07-11 18:15:04.304655: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 510.47.3
2022-07-11 18:15:04.304660: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 510.47.3
2022-07-11 18:15:04.304854: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/facenet/src/lfw.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
return np.array(pairs)
WARNING:tensorflow:From /home/facenet/src/facenet.py:114: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

WARNING:tensorflow:From /home/facenet/src/facenet.py:133: batch_join (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.interleave(...).batch(batch_size) (or padded_batch(...) if dynamic_pad=True).
WARNING:tensorflow:From /home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/input.py:732: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
WARNING:tensorflow:From /home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/input.py:732: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
Model directory: /home/models/facenet/20180402-114759
Metagraph file: model-20180402-114759.meta
Checkpoint file: model-20180402-114759.ckpt-275
2022-07-11 18:15:06.172482: W tensorflow/core/common_runtime/graph_constructor.cc:1526] Importing a graph with a lower producer version 24 into an existing graph with producer version 1087. Shape inference will have run different parts of the graph with different producer versions.
Traceback (most recent call last):
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 500, in _import_graph_def_internal
results = c_api.TF_GraphImportGraphDefWithResults(
tensorflow.python.framework.errors_impl.InvalidArgumentError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "src/validate_on_lfw.py", line 167, in
main(parse_arguments(sys.argv[1:]))
File "src/validate_on_lfw.py", line 75, in main
facenet.load_model(args.model, input_map=input_map)
File "/home/facenet/src/facenet.py", line 383, in load_model
saver = tf.train.import_meta_graph(os.path.join(model_exp, meta_file), input_map=input_map)
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1582, in import_meta_graph
return _import_meta_graph_with_return_elements(meta_graph_or_file,
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1603, in _import_meta_graph_with_return_elements
meta_graph.import_scoped_meta_graph_with_return_elements(
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/meta_graph.py", line 804, in import_scoped_meta_graph_with_return_elements
imported_return_elements = importer.import_graph_def(
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 561, in new_func
return func(*args, **kwargs)
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 404, in import_graph_def
return _import_graph_def_internal(
File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 505, in _import_graph_def_internal
raise ValueError(str(e))
ValueError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

[halfroad]

For those who undergo this problem. I would suggest following actions:

1. Add the directive "import tensorflow.compat.v1 as tf" to the corresponding .py files.
2. Use the arguments to specify the model file and pair.txt with absolute full path, as following,

python3 FaceNet/src/validate_on_lfw.py ../Inventory/Aligned /Users/xxxx/Projects/Inventory/Models/20180402-114759.pb --distance_metric 1 --use_flipped_images --subtract_mean --use_fixed_image_standardization --lfw_pairs /Users/xxxx/Projects//FaceNet/data/pairs.txt

[michealChin]

For those who encounter this problem, please refer to this link
openvinotoolkit/openvino#11078

[HSSH2110]

HELP! I got the same issue yet still can not fix it. I have tried different ways :((

[omkar3102]

same issue cant fix it :(