Add dim check while calculating mIoU&pixAcc for segmentation test #991

chunhanl · 2019-10-18T04:58:32Z

The test.py example mentioned at
segmentation.rst
does not match the arguments mentioned in test.py
should probably be change to
python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval
The dimension doesn't seem right when calculating the pixACC and mIoU
The output is 3D and the target is 2D
The safest way I could think of is to add a new axis if batch dim is missing.
Might have a better way to deal with.

Result after modification (without training)

chunhanl · 2019-10-18T05:12:28Z

Apologies for the new spaces, it was automatically modified by Pycharm IDE

mli · 2019-10-18T05:23:52Z

Job PR-991-1 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-991/1/index.html
Code coverage of this PR: vs. Master:

zhanghang1989 · 2019-10-18T17:13:24Z

docs/model_zoo/segmentation.rst

@@ -31,7 +31,7 @@ Table of pre-trained models for semantic segmentation and their performance.
 The test script :download:`Download test.py<../../scripts/segmentation/test.py>` can be used for
 evaluating the models (VOC results are evaluated using the official server). For example ``fcn_resnet50_ade``::

- python test.py --dataset ade20k --model-zoo fcn_resnet50_ade --eval
+ python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval


why make this change?

Hello,
According in test.py

gluon-cv/scripts/segmentation/test.py

Line 20 in a78965f

def parse_args():

The args defined in parse_args() no longer has args 'model-zoo'

Please add that back instead. Not sure who removed that.

I'm not sure whether this may have conflicts with these settings.

gluon-cv/scripts/segmentation/test.py

Lines 204 to 230 in a78965f

model_prefix = args.model + '_' + args.backbone

if 'pascal' in args.dataset:

model_prefix += '_voc'

withQuantization = True if (args.backbone in ['resnet101']) else withQuantization

elif args.dataset == 'coco':

model_prefix += '_coco'

withQuantization = True if (args.backbone in ['resnet101']) else withQuantization

elif args.dataset == 'ade20k':

model_prefix += '_ade'

elif args.dataset == 'citys':

model_prefix += '_citys'

else:

raise ValueError('Unsupported dataset {} used'.format(args.dataset))

if args.ngpus > 0:

withQuantization = False

if withQuantization and args.quantized:

model_prefix += '_int8'

if not args.deploy:

if args.calibration:

args.pretrained = True

# create network

if args.pretrained:

model = get_model(model_prefix, pretrained=True)

model.collect_params().reset_ctx(ctx=args.ctx)

zhanghang1989 · 2019-10-18T17:14:19Z

gluoncv/utils/metrics/segmentation.py

 # the category -1 is ignored class, typically for background / boundary
+ if len(output.shape) == 3:


batch_intersection_union is supposed to work for a batch with 4D.

OK I'll try to modify from the dataset

chunhanl · 2019-10-18T23:37:39Z

python test.py --dataset ade20k --pretrained --model fcn --backbone resnet50 --eval

python test.py --dataset pascal_voc --pretrained --model fcn --backbone resnet101 --eval

mli · 2019-10-18T23:59:15Z

Job PR-991-2 is done.
Docs are uploaded to http://gluon-vision-staging.s3-website-us-west-2.amazonaws.com/PR-991/2/index.html
Code coverage of this PR: vs. Master:

zhanghang1989 · 2019-10-21T16:27:07Z

scripts/segmentation/test.py

@@ -120,7 +120,7 @@ def test(model, args, input_transform):
 tbar = tqdm(test_data)
 for i, (data, dsts) in enumerate(tbar):
 if args.eval:
- predicts = [pred[0] for pred in evaluator.parallel_forward(data)]
+ predicts = [pred for pred in evaluator.parallel_forward(data)]


This is incorrect. The output of gluoncv segmentation network is always a list

chunhanl added 3 commits October 17, 2019 21:43

Add batch dims for args if missing

b9d6bfa

Add comment

09fc532

Modify command line for segmentation test

c92a1e6

chunhanl changed the title ~~Add dim check for segmentation~~ Add dim check while calculating mIoU&pixAcc for segmentation test Oct 18, 2019

hetong007 requested a review from zhanghang1989 October 18, 2019 08:33

zhanghang1989 reviewed Oct 18, 2019

View reviewed changes

Restore segmentation matrics to origin, remove indexing in test function

43cda24

zhanghang1989 suggested changes Oct 21, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dim check while calculating mIoU&pixAcc for segmentation test #991

Add dim check while calculating mIoU&pixAcc for segmentation test #991

chunhanl commented Oct 18, 2019

chunhanl commented Oct 18, 2019

mli commented Oct 18, 2019

zhanghang1989 Oct 18, 2019

chunhanl Oct 18, 2019 •

edited

zhanghang1989 Oct 18, 2019

chunhanl Oct 18, 2019

zhanghang1989 Oct 18, 2019

chunhanl Oct 18, 2019

chunhanl commented Oct 18, 2019

mli commented Oct 18, 2019

zhanghang1989 Oct 21, 2019

	model_prefix = args.model + '_' + args.backbone
	if 'pascal' in args.dataset:
	model_prefix += '_voc'
	withQuantization = True if (args.backbone in ['resnet101']) else withQuantization
	elif args.dataset == 'coco':
	model_prefix += '_coco'
	withQuantization = True if (args.backbone in ['resnet101']) else withQuantization
	elif args.dataset == 'ade20k':
	model_prefix += '_ade'
	elif args.dataset == 'citys':
	model_prefix += '_citys'
	else:
	raise ValueError('Unsupported dataset {} used'.format(args.dataset))

	if args.ngpus > 0:
	withQuantization = False

	if withQuantization and args.quantized:
	model_prefix += '_int8'

	if not args.deploy:
	if args.calibration:
	args.pretrained = True
	# create network
	if args.pretrained:
	model = get_model(model_prefix, pretrained=True)
	model.collect_params().reset_ctx(ctx=args.ctx)

		# the category -1 is ignored class, typically for background / boundary
		if len(output.shape) == 3:

Add dim check while calculating mIoU&pixAcc for segmentation test #991

Are you sure you want to change the base?

Add dim check while calculating mIoU&pixAcc for segmentation test #991

Conversation

chunhanl commented Oct 18, 2019

chunhanl commented Oct 18, 2019

mli commented Oct 18, 2019

zhanghang1989 Oct 18, 2019

Choose a reason for hiding this comment

chunhanl Oct 18, 2019 • edited

Choose a reason for hiding this comment

zhanghang1989 Oct 18, 2019

Choose a reason for hiding this comment

chunhanl Oct 18, 2019

Choose a reason for hiding this comment

zhanghang1989 Oct 18, 2019

Choose a reason for hiding this comment

chunhanl Oct 18, 2019

Choose a reason for hiding this comment

chunhanl commented Oct 18, 2019

mli commented Oct 18, 2019

zhanghang1989 Oct 21, 2019

Choose a reason for hiding this comment

chunhanl Oct 18, 2019 •

edited