numpy throwing error in batch mode when some images do not contain a face

[markwillowtree]

venv/lib/python3.12/site-packages/facenet_pytorch/models/mtcnn.py", line 444, in select_boxes
selected_boxes = np.array(selected_boxes)
^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (316,) + inhomogeneous part.

I got this error message from this code

    mtcnn = MTCNN(112, device="cuda:0")

    container = av.open(video_path)
    frames = list(container.decode(video=0))
    images = [frame.to_image() for frame in frames]

    mtcnn(images)

It appears to be similar to the one described in this resolved issue:

#206

I made the following changes to lines 444-446 in the file 'models/mtcnn.py'. These modifications ensure that when there are no faces detected in an image, the function returns None.

        if batch_mode:
            selected_boxes = np.array(selected_boxes, dtype=object)
            selected_probs = np.array(selected_probs, dtype=object)
            selected_points = np.array(selected_points, dtype=object)

However, I'm uncertain if this fix might have adverse effects elsewhere in the codebase as I'm not familiar with it.

[florianblume]

I ran into the same problem. My solution is to remove the boxes that are None and delete the corresponding images. I achieved this by returning the images from this function back to forward, like so:

forward function

# Detect faces
batch_boxes, batch_probs, batch_points = self.detect(img, landmarks=True)
# Select faces
if not self.keep_all:
    img, batch_boxes, batch_probs, batch_points = self.select_boxes(
        batch_boxes, batch_probs, batch_points, img, method=self.selection_method,
    )
# Extract faces
faces = self.extract(img, batch_boxes, save_path)

if return_prob:
    return faces, batch_probs
else:
    return faces

and the select_boxes function (omitted everything before if batch_mode because nothing changed there)

if batch_mode:
    sanitized_selected_boxes = []
    sanitized_selected_probs = []
    sanitized_selected_points = []
    for idx, box in enumerate(selected_boxes):
        if box is None:
            if isinstance(imgs, list):
                imgs.pop(idx)
            elif isinstance(imgs, np.ndarray):
                imgs = np.delete(imgs, idx, 0)
            elif isinstance(imgs, torch.Tensor):
                imgs = torch.cat((imgs[:idx], imgs[idx+1:]), 0)
            else:
                raise TypeError("imgs must be a list, np.ndarray, or torch.Tensor")
        else:
            sanitized_selected_boxes.append(box)
            sanitized_selected_probs.append(selected_probs[idx])
            sanitized_selected_points.append(selected_points[idx])
    selected_boxes = np.array(sanitized_selected_boxes)
    selected_probs = np.array(sanitized_selected_probs)
    selected_points = np.array(sanitized_selected_points)
else:
    selected_boxes = selected_boxes[0]
    selected_probs = selected_probs[0][0]
    selected_points = selected_points[0]

return imgs, selected_boxes, selected_probs, selected_points

[beicodewarrior]

Hi, @florianblume

Your code fixed a certain issue when some images don't contain faces. However, it threw a new error when testing one image in the inference script.

mtcnn = MTCNN(
    image_size=160, margin=0, min_face_size=20,
    thresholds=[0.6, 0.7, 0.7], factor=0.709, post_process=True,
    device=device
)

img = Image.open('1.jpg')
img_cropped = mtcnn(img)

This code throws the error like

lib\site-packages\facenet_pytorch\models\utils\detect_face.py", line 359, in extract_face
    margin * (box[2] - box[0]) / (image_size - margin),
TypeError: 'float' object is not subscriptable

Can you check it on your side?

[florianblume]

I would've expected that this doesn't affect a single image because of the if batch_mode: check in the original code. I don't need to process a single image so I'll just use the solution I posted. But I guess it should be an easy fix if you debug the code a bit.

[beicodewarrior]

The returned image value you create seems to pass the if batch_mode:
So it doesn't recognize the single image and batch group. Could you please update your code for that?

[florianblume]

Sorry, I don't have time for that, you'll have to fix it yourself (and you seem to be on the right track).