How to improve real-time face recognition performance and image quality #317

IgorSamer · 2022-12-15T22:12:26Z

IgorSamer
Dec 15, 2022

Hi Vlad!

As you already know, I have an Electron app and came from Amazon Rekognition because I needed a way to run facial recognition in real-time and in offline mode, but now I'm running into some issues - which maybe aren't issues with the code itself - which is why I came to here, to find out if I'm on the right path and also to get tips regarding photography, if possible.

At first glance everything works fine testing on my local machine (even with WASM backend and an Intel Core i3) but I can't get the same satisfactory results with a specific customer (which does not have a GPU) and so I came here again to ask for help on where I can improve my code/camera setup.

I use facial recognition to allow people to enter the gym or not through a turnstile and in this case (which is not so uncommon) it is located right after the entrance door which is where a lot of light comes from and the webcam is on the side (the customer enters and turns to the left or right - on the table where the reception and the computer are - to look at the camera), thus resulting in excessive lighting of the faces and leaving them in many cases with half of the face practically white. The same does not occur at night, where my client said that everything works almost perfectly and the view of the face on the camera is much clearer, that is, my biggest problem is really the natural lighting during the day.

I thought about using the filters for this purpose but they are applied to the whole image and not just where I need to balance, so the result is practically the same. I also tried repositioning the camera a few times, but it didn't help. Do you think adding a light coming from the opposite direction would help with that lighting balance?

And to get a better idea of the content recorded by Human when registering a new face, I started saving photos of all registered faces locally and I was able to see exactly that: registered faces during the day are extremely illuminated and registered faces at night are much more visible, but something that made me a little concerned regardless of the time of day, was that all the photos seem to be of very low quality, some even being pixelated. I believe that this bad quality affects the comparison of faces, right? If you want, I can give you some examples of captured photos, I just won't post them here for permission and privacy reasons.

All photos are 192x192 pixels and I don't know how to improve that - I understand that it only returns the extracted face from the image, but it worried me anyway. The webcam used is Full HD and nowhere in my code do I restrict the width or height of the camera, so how can I improve this resolution?

My code is this (in a simplified way):

<body>
    <div class="container">
        <div class="inner">
            <canvas id="canvas"></canvas>
        </div>
    </div>

    <script type="text/javascript">
        document.addEventListener('DOMContentLoaded', () => {
            const minSameFaceDetectionTime = 10
            const maxFaceAngleRange = 20
            const maxDistance = 30

            const humanConfig = {
                debug: false,
                backend: 'wasm',
                deallocate: true,
                modelBasePath: '../models',
                filter: { enabled: true, equalization: true },
                face: {
                    enabled: true,
                    detector: { return: true, rotation: true, maxDetected: 1, minConfidence: 0.8 },
                    emotion: { enabled: false },
                    iris: { enabled: true }
                },
                body: { enabled: false },
                hand: { enabled: false },
                gesture: { enabled: false }
            }
            const human = new Human.Human(humanConfig)

            human.draw.options.font = `bold small-caps 18px "Rubik"`
            human.draw.options.drawLabels = false
            human.draw.options.drawPolygons = false
            human.draw.options.drawGaze = false

            const canvas = document.getElementById('canvas')
            const context = canvas.getContext('2d')
            const tempCanvas = document.createElement('canvas')

            let faces = []
            let descriptors = []

            const getSecondsDiff = (date) => {
                if (date) return Math.round((new Date() - date) / 1000)

                return minSameFaceDetectionTime + 1
            }

            const getAngleInDegrees = (angle) => angle * 180 / Math.PI

            const setUserLastDetectionTime = (userId, time) => {
                const userFaces = faces.filter((face) => face.id == userId)

                for (let face of userFaces) face.lastDetectionTime = time
            }

            const initCamera = async () => {
                try {
                    await human.webcam.start({ id: device })

                    human.video(human.webcam.element)

                    canvas.width = human.webcam.width
                    canvas.height = human.webcam.height

                    drawLoop()
                } catch (error) {
                    console.error(error)
                }
            }

            const drawLoop = async () => {
                human.draw.canvas(human.webcam.element, canvas)

                const interpolated = human.next()
                const face = interpolated.face[0]
                const angle = face?.rotation.angle
                const match = human.match(face?.embedding, descriptors)
                const drawOptions = {}

                // Using the angles of the face to define whether it's centered or not instead of using gestures for that - I think that way it would be less costly for the CPU, right?
                const faceAngleInRange = Math.abs(getAngleInDegrees(angle?.yaw)) <= maxFaceAngleRange && Math.abs(getAngleInDegrees(angle?.pitch)) <= maxFaceAngleRange
                
                // Setting the maximum distance to ignore people in the background and only consider those close to the camera/turnstile
                const faceInDistanceRange = face?.iris <= maxDistance
                let label

                if (match.similarity >= minSimilarity) {
                    const user = faces[match.index]

                    if (user) {
                        label = user.name

                        if (faceInDistanceRange) {
                            if (faceAngleInRange) {
                                drawOptions.color = 'green'

                                const lastDetectionTime = getSecondsDiff(user.lastDetectionTime)

                                if (lastDetectionTime > minSameFaceDetectionTime) {
                                    setUserLastDetectionTime(user.id, new Date())

                                    console.log('Allow user access')
                                } else {
                                    label += ` (${10 - lastDetectionTime})`
                                }
                            }
                        }
                    }
                } else {
                    drawOptions.color = 'red'
                }

                if (face) {
                    let text = label || 'Unknown'

                    if (!faceInDistanceRange) {
                        text += ' (too far)'
                    } else if (!faceAngleInRange) {
                        text += ' (non-centralized)'
                    }

                    const [x, y] = face.box

                    context.strokeStyle = '#fff'
                    context.strokeText(text, x, y - 10)
                    context.fillText(text, x, y - 10)
                }

                await human.draw.face(canvas, interpolated.face, drawOptions)

                setTimeout(drawLoop, 30)
            }

            const registerFace = async (data) => {
                const faceDetected = human.result.face[0]

                if (!faceDetected) {
                    console.log('No faces were detected.')

                    return
                }

                if (faceDetected.iris > maxDistance) {
                    console.log('The face is too far from the camera.')

                    return
                }

                const angle = faceDetected.rotation.angle

                if (Math.abs(getAngleInDegrees(angle.yaw)) > maxFaceAngleRange) {
                    console.log('The face is not horizontally centered.')

                    return
                }

                if (Math.abs(getAngleInDegrees(angle.pitch)) > maxFaceAngleRange) {
                    console.log('The face is not vertically centered.')

                    return
                }

                const face = { id: data.userId, name: data.firstname, embedding: faceDetected.embedding }

                try {
                    await human.tf.browser.toPixels(faceDetected.tensor, tempCanvas)

                    const picture = tempCanvas.toDataURL('image/jpeg', 1).replace(/^data:image\/jpeg;base64,/, '')

                    human.tf.dispose(faceDetected.tensor)

                    // here I send 'picture' to main process to save the image as file and then:

                    face.lastDetectionTime = new Date(Date.now() - 7000)

                    faces.push(face)
                    descriptors.push(face.embedding)

                    // Setting the first detection of the face that was just registered for 3 seconds from now
                    setUserLastDetectionTime(face.id, face.lastDetectionTime)
                } catch (error) { }
            })

            initCamera()
        })
    </script>
</body>

The webcam I recommend for my clients is the Logitech Full HD Pro Stream C922 and that specific client's CPU is an Intel Core i5. For now I'm having to keep Amazon Rekognition in this gym, because, although the conditions are exactly the same, Rekognition manages to register and recognize faces normally. However I want to test all the possibilities to be able to use Human - I was delighted with this project.

Do you believe that there would be improvements in facial recognition if in these cases of lack of GPU I stopped using Human in real time to do the same as Rekognition and require less processing? (take the photo in the highest possible quality and recognize it based on the faces already saved)

I'm available to answer any questions and thank you again for the amazing project!

Answered by vladmandic

Dec 16, 2022

Do I need to make this value explicit anyway along with filter: { enabled: true, equalization: true } for it to take effect?

if you're using human v3, filter.enabled and filter.autobrightness are enabled by default (unless you have a bad browser which doesn't support real-time filtering, but that's another story), so need to enable it explicitly.
filter.equalization is a different algorithm and its disabled by default - you can try if it instead of autobrightness, but not both at the same time.

In that case I will instruct the staff to turn this light off at night. The important thing is to be able to recognize faces in daylight as well.

its ok if scene is fully lit or its a bit darke…

View full answer

vladmandic · 2022-12-15T22:45:03Z

vladmandic
Dec 15, 2022
Maintainer

I tried repositioning the camera a few times, but it didn't help. Do you think adding a light coming from the opposite direction would help with that lighting balance?

yes. most cameras (even cheap webcams) have some auto-adjustment,
so if you add additional light so there is less difference between light/dark, it will lower overall brightness.

there is also a new feature added very recently to human: config.filter.autoBrightness, you can enable that and see if it helps.
if images have natural light, it will help balance out images taken during day and during night.
but if night photos have very high contrast because of a single directional light, nothing will really help.

I believe that this bad quality affects the comparison of faces, right

of course...

Do you believe that there would be improvements in facial recognition if in these cases of lack of GPU I stopped using Human in real time to do the same as Rekognition and require less processing? (take the photo in the highest possible quality and recognize it based on the faces already saved)

This doesn't sound it has to be a perfect real-time, so why not?

All photos are 192x192 pixels and I don't know how to improve that

If you want to store higher resolution, you can crop original photo using face box coordinates yourself, its pretty trivial - but that would be only for visual display in app, nothing to do with recognition.

if you use face.tensor, its always 192x192 because that is what model is trained on.

1 reply

IgorSamer Dec 15, 2022
Author

so if you add additional light so there is less difference between light/dark, it will lower overall brightness.

Got it, I think what I really need is a consultancy with a photographer to get experience for these cases haha

there is also a new feature added very recently to human: config.filter.autoBrightness, you can enable that and see if it helps.

Okay, I didn't include this property because I saw at

human/src/config.ts

Line 355 in cb36466

autoBrightness: true,

that it is true by default. Do I need to make this value explicit anyway along with filter: { enabled: true, equalization: true } for it to take effect?

but if night photos have very high contrast because of a single directional light, nothing will really help.

In that case I will instruct the staff to turn this light off at night. The important thing is to be able to recognize faces in daylight as well.

If you want to store higher resolution, you can crop original photo using face box coordinates yourself, its pretty trivial - but that would be only for visual display in app, nothing to do with recognition.

if you use face.tensor, its always 192x192 because that is what model is trained on.

Excellent! I thought it was something that would influence recognition and not something about the model itself. Anyway, I'll save the images with the original size, since I intend to keep them stored in case I need to re-register them at some point.

Another question is: if the person wears glasses, is it recommended to take them off to register the face/perform facial recognition or does this not influence that much?

vladmandic · 2022-12-16T00:01:39Z

vladmandic
Dec 16, 2022
Maintainer

Do I need to make this value explicit anyway along with filter: { enabled: true, equalization: true } for it to take effect?

if you're using human v3, filter.enabled and filter.autobrightness are enabled by default (unless you have a bad browser which doesn't support real-time filtering, but that's another story), so need to enable it explicitly.
filter.equalization is a different algorithm and its disabled by default - you can try if it instead of autobrightness, but not both at the same time.

In that case I will instruct the staff to turn this light off at night. The important thing is to be able to recognize faces in daylight as well.

its ok if scene is fully lit or its a bit darker (autobrightness will take care of that), worst case is if its lit just from one side which creates large contrasts.

if the person wears glasses, is it recommended to take them off to register the face/perform facial recognition or does this not influence that much?

right now, it has quite a big impact - much bigger than i'd like and its something i'm looking to improve in the future.

on a side-note, human v3 supports completely new module for face descriptors - insightface,
but its disabled by default as its a bit larger than current one and its not a common thing that high-precision is required.

also, descriptors captured with new module are not compatible with old one, so you'd have to retrain database.

if you want to experiment, you can take a look

notes and models: https://github.com/vladmandic/insightface
example with configuration: https://github.com/vladmandic/insightface/blob/main/src/index.ts

(no changes needed in human, you only need to download the model and enable it in human config.)

2 replies

IgorSamer Dec 17, 2022
Author

its ok if scene is fully lit or its a bit darker (autobrightness will take care of that), worst case is if its lit just from one side which creates large contrasts.

Great! I'll work on avoiding large contrasts, because that's exactly the problem I'm facing right now.

right now, it has quite a big impact - much bigger than i'd like and its something i'm looking to improve in the future.

Nice, that would help a lot!

on a side-note, human v3 supports completely new module for face descriptors - insightface,
but its disabled by default as its a bit larger than current one and its not a common thing that high-precision is required.

I will definitely take a look!

Thank you so much for your attention and quick replies!

Lamarcke Jul 2, 2023

Hello. I'm sorry to bring back an older topic, but i'm building a project which is similar to the one OP mentioned. I'm having a problem where some faces are being mislabeled as others (descriptor similarity is higher for the wrong face). I've experimented a bit and it seems related to the issues with huge light contrasts too.

Do you think insightface would have a better precision in this regard? The ideal situation would be that wrong faces would have very low similarity (<0.5, currently, some are only lower than the correct one by a very small margin), so i assume having a higher descriptor precision would help with that?

For my use case, it would be better to have no match than it is to match with the incorrect face. Thank you for your work.

FICGAN-classification · 2024-03-27T15:02:10Z

FICGAN-classification
Mar 27, 2024

Improved Time-Saving Face Detection in Video
https://ieeexplore.ieee.org/document/10438811

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to improve real-time face recognition performance and image quality #317

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to improve real-time face recognition performance and image quality #317

IgorSamer Dec 15, 2022

Replies: 3 comments · 3 replies

vladmandic Dec 15, 2022 Maintainer

IgorSamer Dec 15, 2022 Author

vladmandic Dec 16, 2022 Maintainer

IgorSamer Dec 17, 2022 Author

Lamarcke Jul 2, 2023

FICGAN-classification Mar 27, 2024

IgorSamer
Dec 15, 2022

Replies: 3 comments 3 replies

vladmandic
Dec 15, 2022
Maintainer

IgorSamer Dec 15, 2022
Author

vladmandic
Dec 16, 2022
Maintainer

IgorSamer Dec 17, 2022
Author

FICGAN-classification
Mar 27, 2024