left-right flip

[jutanke]

It seems like left and right are flipped in your 3D representation, e.g. the following code

human-motion-prediction/src/viz.py

Line 25 in c9a2774

self.LR = np.array([1,1,1,0,0,0,0, 0, 0, 0, 0, 0, 0, 1, 1, 1], dtype=bool)

seems to conflict with the human 3.6m matlab code

switch part
        case 'rootpos'
          joints = 1;
        case 'rootrot'
          joints = 1;
        case 'leftarm'
          joints = 18:24;% p/p2/a fine
        case 'rightarm'
          joints = 26:32;% p/p2/a fine
        case 'head'
          joints = 14:16;% p/p2/a fine
        case 'rightleg'
          joints = 2:6;% p/p2/a fine
        case 'leftleg'
          joints = 7:11;% p/p2/a fine
        case 'upperbody'
          joints = [14:32];% p/p2/a fine
        case 'arms'
          joints = [16:32];% p/p2/a fine
        case 'legs'
          joints = 1:11;% p/p2/a fine
        case 'body'
          joints = [1 2 3 4 7 8 9 13 14 15 16 18 19 20 26 27 28];% p/p2/a fine
        otherwise
          error('Unknown');
      end

.
The plot below shows the original 3D ground truth location ('Ground Truth') from human3.6m, the particular frame of the video and the 3D plot in expmap format, transformed to 3D joint coordinates using your forward kinematics function, provided with your repository ('ExpMaps').

Is there an intention behind the flip?

[una-dinosauria]

Hi @jutanke,

Nice catch. There seems to be a reflection about the x axis. This seems quite strange to me to be honest, as it's hard for me to image a preprocessing that would accidentally flip the coordinates.

I am not sure what the visualization function has to do with this. Could you please elaborate?
I probably wrote that visualization code without looking at the reference matlab, so I just wanted to make sure L/R were internally consistent -- changing this, however, would probably only change colours, not the reflection.

[jutanke]

Hi @una-dinosauria ,
I just pointed out the visualization as I imagine you chose the joint ids by visual inspection. I'd guess that suggests that the flip is either caused by the forward kinematics or is already part of the exponential map data. What I am "worried" about is that there are other not-so-easy-to-spot inconsistencies with the ground truth. I will try to investigate that a bit further these days and report if something interesting comes up.
cheers

[LittleFlyFish]

I am just trying to find this figure today! So nice of you, it is very useful for my project.

[LittleFlyFish]

It seems like left and right are flipped in your 3D representation, e.g. the following code

human-motion-prediction/src/viz.py

Line 25 in c9a2774

self.LR = np.array([1,1,1,0,0,0,0, 0, 0, 0, 0, 0, 0, 1, 1, 1], dtype=bool)

seems to conflict with the human 3.6m matlab code

switch part
        case 'rootpos'
          joints = 1;
        case 'rootrot'
          joints = 1;
        case 'leftarm'
          joints = 18:24;% p/p2/a fine
        case 'rightarm'
          joints = 26:32;% p/p2/a fine
        case 'head'
          joints = 14:16;% p/p2/a fine
        case 'rightleg'
          joints = 2:6;% p/p2/a fine
        case 'leftleg'
          joints = 7:11;% p/p2/a fine
        case 'upperbody'
          joints = [14:32];% p/p2/a fine
        case 'arms'
          joints = [16:32];% p/p2/a fine
        case 'legs'
          joints = 1:11;% p/p2/a fine
        case 'body'
          joints = [1 2 3 4 7 8 9 13 14 15 16 18 19 20 26 27 28];% p/p2/a fine
        otherwise
          error('Unknown');
      end

.
The plot below shows the original 3D ground truth location ('Ground Truth') from human3.6m, the particular frame of the video and the 3D plot in expmap format, transformed to 3D joint coordinates using your forward kinematics function, provided with your repository ('ExpMaps').

Is there an intention behind the flip?

Are you sure about the label of joints number in the figure? From the code to visualize data, I see that 👍

parent = np.array([0, 1, 2, 3, 4, 5, 1, 7, 8, 9,10, 1,12,13,14,15,13,
17,18,19,20,21,20,23,13,25,26,27,28,29,28,31])-1

Because 1 has appearance three times, 13 two times so I think the hip is 1 and chest is 13, or hip is 0 and chest is 12. It cannot be hip mark 0 and chesht 13, it doesn't match this code

[jutanke]

Hi,
I just take the exp map data taken from here, as explained here and then parse the n x 99 dim into 3D joint coordinates using forward kinematics as done here:

human-motion-prediction/src/forward_kinematics.py

Lines 174 to 178 in c9a2774

	xyz_gt, xyz_pred = np.zeros((nframes_gt, 96)), np.zeros((nframes_pred, 96))
	for i in range( nframes_gt ):
	xyz_gt[i,:] = fkl( expmap_gt[i,:], parent, offset, rotInd, expmapInd )
	for i in range( nframes_pred ):
	xyz_pred[i,:] = fkl( expmap_pred[i,:], parent, offset, rotInd, expmapInd )

. This yields n x 32 x 3 as 3D joint coordinates which you can then simply plot. The number is the respective index. I am not super familiar with the code base yet but the code you posted

parent = np.array([0, 1, 2, 3, 4, 5, 1, 7, 8, 9,10, 1,12,13,14,15,13,17,18,19,20,21,20,23,13,25,26,27,28,29,28,31])-1

seems to be taken from matlab as there is a -1 at the end.
Joint 0 and joint 11 (when starting with 0) seem to both be the root joint based at the centre of the hip in the ground-truth data as well as in the expmap data.
Cheers

[jutanke]

Update
The 3D representation extracted from the exponential maps via forward kinematics seems to have some other interesting properties.

I align the ground-truth pose (poses from the original dataset and not converted from the exp maps) by first translating the whole pose so that the hip centre is at (0, 0, 0), then rotate around the z-axis so that the person faces forward in the positive y-direction ("facing" means: the 2d normal in the xy-plance from the vector [left hip - right hip]), then rotate around the y-axis to ensure that the left and right hip both touch the x-axis.

In the pictures below the orange/green person is from the original ground truth points while the blue/red person is the pose extracted by forward kinematics from the exponential maps. It seems that the poses generated by the forward kinematics lean backwards quite significantly. I was first thinking that there might be some constant rotation difference but that is also not the case: in the last plot I plot the the radians along the x-axis needed to match the two poses and it is definitively not constant.

@una-dinosauria do you think that there are issues with the forward kinematics or are there other things at play that I have not taken into account?

All plots below where generated from the second sub-action video "Walking" from actor "S1".

(I flip left and right so that the extracted poses have the correct left/right sides)

[una-dinosauria]

Hi @jutanke,

Thanks for your great work tracking down this issue, and sorry for the delay in responding; I was very busy with a deadline.

I suspect this is probably my fault. I try to be pretty careful when I port code to make these transformations, but it seems like this time I really did mess something up.
Another user noticed errors in my forward_kinematics code -- #23 (comment) -- which is likely the source of error.

You may want to look into that function and the corresponding matlab reference code and that may fix the issue. I'll be super grateful if you can submit a fix as well.

Cheers,