headless version

[chamantonio]

Hi! I'm trying to create a headless version of this by replacing PyOpenGL with ModerlGL. I'm planning to use shaders to transfer the texture to the facial 3d model. However it's difficult since I'm new to 3d space & I'm only familiar with DLIB. I hope you can answer these questions abt the parameters you used so I can understand more.

1. What is textureCoords? like how does it relate to a generic Candide.obj 3D model & the candide mesh? Can I use the candide.obj instead of the candide.npz?
   I understand that shapes2D is the list of facial keypoints for all faces detected.
2. What is modelParams?

Thank you so much!

[MarekKowalski]

Hi,

Here it goes:

1. textureCoords are the coordinates of the vertices of the candide model in the texture image. The texture image is an image of the face that is being overlaid on top of the face seen by the camera.
2. The candide.npz file contains several things: the mean 3D model of a face, the blendshapes, the mesh, the correspondences between the 2D landmarks and the vertices of the model. The candide.obj file cannot store all that information.
3. Yes, that is what shapes2D is.
4. modelParams contain the weights of the blendshapes of the candide model and the pose of the head, which is used to render the face.

Hope that helps, tell me if you have more questions.

Marek

[chamantonio]

Thank you! Will try to work with this info.

[chamantonio]

Hi @MarekKowalski I'm trying to recreate these lines w/o the opengl dependency. I can't seem to find a good explanation of what's going on here? Please help me understand what the functions are doing. I put in comments the explanations I found but I really can't visualize what glTexCoord2fv & glVertex3fv are doing.

for triangle in self.mesh:
           
            for vertex in triangle:
               
                #Specify s, t, r, and q texture coordinates
                glTexCoord2fv(self.textureCoords[:, vertex])
               
                #Specify x, y, z, and w coordinates of a vertex.
                glVertex3fv(shape3D[:, vertex])
               

Thanks again in advance!

[MarekKowalski]

Hi,

The first line specifies the location of a vertex in the texture and the second line specifies the 3D position of the corresponding vertex. Those two lines are part of a larger section of code:

glBegin(GL_TRIANGLES)
        for triangle in self.mesh:
            for vertex in triangle:
                glTexCoord2fv(self.textureCoords[:, vertex])
                glVertex3fv(vertices[:, vertex])
            
glEnd() 

This section of code specifies the coordinates of triangle vertices in texture and 3D space. The self.mesh array consists of triples of indices that specify which verices of vertices form the triangles.

Does that explain it?

Marek

[chamantonio]

Thank you so much Marek! I studied it further. So, that section of the code does the delaunay triangulation for both textureCoords & vertices, makes up for 2 meshes of 175 triangles, 1 2D(texture) & 1 3D(camera image shape3d). However, I'm lost to what happened to these specified meshes.

I have an experience face morphing & face swapping in plain 2D approach. It also involves delaunay triangulation then warps the two 2d image in piecewise affine manner.

I can't seem to find a function in this code that does the warping counterpart. Where do you do the actual warping of these 2D & 3D mesh to arrive at the face-swapped output (data& renderedImg)? If the process to arrive does not involve warping then what happens to these meshes after being specified for OpenGL?

Thanks again. You've done so much help.

Cham

[MarekKowalski]

Hi,

There is no delaunay triangulation being performed in the code. The structure of the mesh is provided by the candide model itsef.

Instead of the warping, which you know from the 2D approach, the face mesh is rendered (with the texture from the source image) using OpenGL. The rendering step replaces the warping.

Marek

[chamantonio]

I already achieved the result that I want but I just used warping instead of rendering.
Thanks a lot Marek!

[ak9250]

@chamantonio can you please share the headless version, i am trying to set it up in colab, can share the colab if that works

[Turacbey]

I already achieved the result that I want but I just used warping instead of rendering.
Thanks a lot Marek!

Hey, could you please share the headless version? I will very appreciate if you can.

[GZaccaroni]

Has anyone succeeded in setting it up in google colab?
Thank you!

[ksamrouth]

could you find anything?