Attention and iris augmentations

[saeidn95]

Here are a couple of question about the mesh network argumentation.

a. the attention augmentation which is disabled by default (

human/src/face/facemesh.ts

Line 115 in 1114014

rawCoords = await attention.augment(rawCoords, results); // augment iris results using attention model results

), augments the 468 points from the mesh with 232 additional points that are produced by the five additional set of outputs to a total of 700. These additional 232 points do not have z-coordinates and get their z values averaged from the set of corresponding points from the 468 mesh outputs.

I might be wrong but, but my understanding of the attention is to replace points around the eyes and lips from the mesh output with the more accurate points from the five attention output. If I am correct I would expect replacement and not addition (except of the 5+5 iris points). This seems to be how the default iriis attention network work works.(

human/src/face/facemesh.ts

Line 117 in 1114014

rawCoords = await iris.augmentIris(rawCoords, face.tensor, inputSize, config); // run iris model and augment results

)

With iris augmentation on the other hand, only adds 10 new points iris points and replaces old mesh points with the new ones. Shouldn't we do the same thing with attention?

b. Where do i get the attention network in tflite/onnx format?

[vladmandic]

yes, we should replace points and not add them, but i never liked how mediapipe implemented augmentation to start with so i never went too far with it.

the issue is that augmentation network is not really single network, its basically pipe of multiple models connected using proprietary functions - and those functions are only implemented as unofficially software functions for tfjs-webgl backend, they don't exist in actual tensorflow.

for that reason, attention model cannot run in nodejs (since functions dont exist in tfjs-node) or tflite.

reality is, mediapipe never intended attention model to run outside of mediapipe backend.

[saeidn95]

Well explained!

When I plot the both original 468 points from mesh model output and the corrected additional 222 from the five set of attention outputs, for the points around the eyes and lips, the corrected attention points are lot closer to their intended positions. Unfortunately, these additional points do not z coordinates. I wonder if I can borrow the z coordinate from their corresponding original 468 points. I don't think using the averages would be very useful.

The iris network that you use comes with 3 coordinates, which is good.

Can you please point me to origin of the iris network in Human, and where I may get network in tflite/onnx format?

[vladmandic]

if you do map attention points back to full face mesh, it would be great - and i don't see a major problem reusing z-coordinates.
for iris, this is a good starting point: https://github.com/PINTO0309/PINTO_model_zoo/tree/main/049_iris_landmark

[vladmandic]

and this is a conversation on those functions in attention pipeline i've mentioned: PINTO0309/PINTO_model_zoo#143

[saeidn95]

Great! Thank you,

a. Looking at your comment in PINTO0309/PINTO_model_zoo#143
"fyi, i've just added facemesh attention to https://vladmandic.github.io/human,
works nice but it is ~%25 slower than facemesh + iris models combined"

I find this comment interesting. Whatever way we look at it, from this comment I conclude that it is best to avoid the use of "facemesh+attention" and instead use "facemesh" and "iris". With iris we also get the a z component for the iris points, and things also run faster too! So, there is no point to worry about the "facemesh+attention"

b. I also notice that you make double correction to the z components of the iris network,
by not just by replacing old with new, but adding half of the old values the new values ( newCoords[index][2] + rawCoords[originalIndices[j]][2]) / 2), (

human/src/face/iris.ts

Line 49 in 1114014

(newCoords[index][2] + rawCoords[originalIndices[j]][2]) / 2,

).

Did you arrive at this double correction by experimentation?

[vladmandic]

that's just a trick to reduce jumps/flicker when processing real time video input such as webcam