Close eyes situation

[Bornblack]

@TimoBolkart
Hi,
Thanks for your great work. The objective weights you provided can make the fitting result good most of the time. But when the eyes are closed, the eyes of the fitting result are open. Although I try to tune the weights and add the eye distance objective, the result is still eyes-open.
Here are the implementation of the eye distance objective.

def eyes_distance(lmk):
    # left eye:  [21,22], [25,24] - 1
    # right eye: [27,28], [31,30] - 1
    eye_up = lmk[[20,21,26,27],:]
    eye_bottom = lmk[[24,23,30,29],:]
    dis = ch.array((eye_up - eye_bottom))
    return dis
    
# -----------------------------------------------------------------------------

def eyes_landmark_error_3d( mesh_verts, mesh_faces, lmk_3d, lmk_face_idx, lmk_b_coords, weight=1.0 ):
    """ function: 3d landmark error objective
    """
    # select corresponding vertices
    v_selected = mesh_points_by_barycentric_coordinates( mesh_verts, mesh_faces, lmk_face_idx, lmk_b_coords )
    lmk_num  = lmk_face_idx.shape[0]
    # an index to select which landmark to use
    lmk_selection = np.arange(0,lmk_num).ravel() # use all
    # residual vectors
    lmk3d_obj = weight * (eyes_distance(v_selected[lmk_selection]) - eyes_distance(lmk_3d[lmk_selection]) )

    return lmk3d_obj
# -----------------------------------------------------------------------------

Any help would be appreciated!

[wuneng]

Maybe high regularization on the expression parameters.

[byfron]

AFAIK the FLAME model does not have a blinking blendshape, so you may have to include custom shapes to get it to blink.
[edited] Ok I just checked the publication and they claim that the model does blink. Maybe as @wuneng mentioned, try to reduce regularization. It may also be that the blinking shape is coupled with other shapes (if the original dataset was biased) and your fit can only blink jointly with other shapes. You can try to visualize/identify the blinking shape, and try to optimize only with the one (or the subset) that produce the blinks.

[TimoBolkart]

FLAME in the public form has no dedicated blendshapes to add eyeblinks but by optimizing the expression parameters, FLAME is able to blink. Commonly, a too high regularization of the expression parameters prevent that as suggested by @wuneng. An alternative option is to use the additional eyeblink blenshapes that we share in the metrical-tracker repository.