🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

karventhan13 · 2024-04-20T09:06:51Z

What's happening?

🐛 [^4.0.0-beta.16] - Camera is not aligned properly when using useSkiaFrameProcessor but it's work with

Reproduceable Code

return (
    <SafeAreaView style={styles.container}>
      {!hasPermission && <Text>No Camera Permission.</Text>}
      {hasPermission && device != null && (
        <Camera
          style={StyleSheet.absoluteFill}
          device={device}
          isActive={true}
          photo={true}
          video={true}
          format={format}
          onError={(err)=> { console.log("err **",err)}}
          fps={fps}
          enableFpsGraph={true}
          frameProcessor={frameProcessor}  
          videoHdr={format.supportsVideoHdr}
          photoHdr={format.supportsPhotoHdr}
          enableZoomGesture={false}
        />
      )}
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
  },
});

Relevant log output

none

Camera Device

none

Device

vivo y21

VisionCamera Version

[^4.0.0-beta.16]

Can you reproduce this issue in the VisionCamera Example app?

I didn't try (⚠️ your issue might get ignored & closed if you don't try this)

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

mrousavy · 2024-04-21T21:47:32Z

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

mrousavy · 2024-04-21T22:03:27Z

Maybe play around with useSkiaFrameProcessor hook, specifically the render() function in there. If you can figure out what's wrong, please submit a PR.
I'm on vacation this week

Space6188 · 2024-04-22T13:40:50Z

This happened when you using resizeMode={'contain'}

Try to delete this from Camera props

mrousavy · 2024-04-22T13:57:43Z

@Space6188 does it properly display without the resizeMode?

mrousavy · 2024-04-22T13:58:00Z

I can also reproduce this when using resizeMode="cover"...

mrousavy · 2024-04-22T14:00:27Z

Try playing around with the code from useSkiaFrameProcessor:

react-native-vision-camera/package/src/skia/useSkiaFrameProcessor.ts

Lines 165 to 175 in 3ff02cc

 'worklet' 

 // rotate canvas to properly account for Frame orientation 

 canvas.save() 

 const rotation = getRotationDegrees(frame.orientation) 

 canvas.rotate(rotation, frame.width / 2, frame.height / 2) 

 // render the Camera Frame to the Canvas 

 if (paint != null) canvas.drawImage(image, 0, 0, paint) 

 else canvas.drawImage(image, 0, 0) 

 // restore transforms/rotations again 

 canvas.restore()

Maybe we're doing something wrong.

Also, the ImageProxy (which is what the native Frame type holds) contains a getSensorToBufferTransformMatrix() method - maybe we can use this Matrix to transform the Frame safely to view dimensions? cc @rodgomesc if you wanna look into this.

rodgomesc · 2024-04-22T14:02:54Z

I will never understand why people ignore the fact that I need log output. I will write a bot in the future to auto-close issues that don't have valid log output.

then you will have zero issues in vc 😅

rodgomesc · 2024-04-22T14:04:37Z

i tried all resizeMode possibilities, also all fit possibilities, the problem is not there

mrousavy · 2024-04-22T14:05:50Z

then it's probably the sensor to buffer transform matrix.

Space6188 · 2024-04-22T15:18:09Z

@Space6188правильно ли он отображается без resizeMode?

No,my bad,sorry, it is not working too

rodgomesc · 2024-04-29T01:51:55Z

i think that i found a fix for this, should send a pr some time tomorrow

AndyRC6 · 2024-04-29T17:51:37Z

In case it helps - I am able to reproduce this by passing in the format prop with a custom videoResolution to the camera. Removing the format prop seems to fix the issue.

Edit: Actually, removing format only fixes the white bar that appears to the left of the preview, and moves it down to the bottom. Which is less noticeable as it looks like that's just where the camera preview ends. But in my case, the preview should take up the entire screen, but doesn't when using the Skia frame processor.

rodgomesc · 2024-04-30T02:53:52Z

it seems that canvas.drawImage draws the image with a default clip and matrix that are messed by the canvas rotation somehow? that’s confusing since if we use canvas.drawImageRect with the same src/dst props it draws correctly! in my tests it seem to solve the issue

diff --git a/package/src/skia/useSkiaFrameProcessor.ts b/package/src/skia/useSkiaFrameProcessor.ts
index 3feb336..fd69ca1 100644
--- a/package/src/skia/useSkiaFrameProcessor.ts
+++ b/package/src/skia/useSkiaFrameProcessor.ts
@@ -6,8 +6,34 @@ import { VisionCameraProxy, wrapFrameProcessorWithRefCounting } from '../FramePr
 import type { DrawableFrameProcessor } from '../types/CameraProps'
 import type { ISharedValue, IWorkletNativeApi } from 'react-native-worklets-core'
 import { WorkletsProxy } from '../dependencies/WorkletsProxy'
-import type { SkCanvas, SkPaint, SkImage, SkSurface } from '@shopify/react-native-skia'
+import { type SkCanvas, type SkPaint, type SkImage, type SkSurface, ColorType, AlphaType, Skia } from '@shopify/react-native-skia'
 import { SkiaProxy } from '../dependencies/SkiaProxy'
+import { InteractionManager } from 'react-native'
+
+// Mocked Frame Data for testings
+const mockedImageWidth = 1920
+const mockedImageHeight = 1080
+
+const pixels = new Uint8Array(mockedImageWidth * mockedImageHeight * 4)
+pixels.fill(255)
+let i = 0
+for (let x = 0; x < mockedImageWidth; x++) {
+  for (let y = 0; y < mockedImageHeight; y++) {
+    pixels[i++] = (x * y) % 255
+  }
+}
+
+const data = Skia.Data.fromBytes(pixels)
+const image = Skia.Image.MakeImage(
+  {
+    width: mockedImageWidth,
+    height: mockedImageHeight,
+    alphaType: AlphaType.Opaque,
+    colorType: ColorType.RGBA_8888,
+  },
+  data,
+  mockedImageWidth * 4,
+)!
 
 /**
  * Represents a Camera Frame that can be directly drawn to using Skia.
@@ -155,7 +181,7 @@ export function createSkiaFrameProcessor(
 
     // Convert Frame to SkImage/Texture
     const nativeBuffer = (frame as FrameInternal).getNativeBuffer()
-    const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
+    // const image = Skia.Image.MakeImageFromNativeBuffer(nativeBuffer.pointer)
 
     return new Proxy(frame as DrawableFrame, {
       get: (_, property: keyof DrawableFrame) => {
@@ -165,11 +191,36 @@ export function createSkiaFrameProcessor(
             'worklet'
             // rotate canvas to properly account for Frame orientation
             canvas.save()
-            const rotation = getRotationDegrees(frame.orientation)
-            canvas.rotate(rotation, frame.width / 2, frame.height / 2)
-            // render the Camera Frame to the Canvas
-            if (paint != null) canvas.drawImage(image, 0, 0, paint)
-            else canvas.drawImage(image, 0, 0)
+
+            const rotationAngle = 0 //getRotationDegrees('portrait')
+            const { width: frameWidth, height: frameHeight } = getPortraitSize(frame)
+
+            // center of the canvas
+            const centerX = frameWidth / 2
+            const centerY = frameHeight / 2
+
+            const currentPaint = paint ?? Skia.Paint()
+
+            // only draw the rect by 40% of the frame size so we can vizualize the whole thing to debug :)
+            const reductionFactor = 0.4
+            const rectWidth = frameWidth * reductionFactor
+            const rectHeight = frameHeight * reductionFactor
+
+            // apply rotation around the center of the canvas
+            canvas.rotate(rotationAngle, centerX, centerY)
+
+            const rectX = centerX - rectWidth / 2
+            const rectY = centerY - rectHeight / 2
+
+            const srcRect = Skia.XYWHRect(0, 0, image.width(), image.height())
+            // Define the destination rectangle on the canvas
+            const dstRect = Skia.XYWHRect(rectX, rectY, rectWidth, rectHeight)
+            // Draw the image on the canvas
+
+            canvas.drawImageRect(image, srcRect, dstRect, currentPaint)
 
             // restore transforms/rotations again
             canvas.restore()
@@ -185,7 +236,7 @@ export function createSkiaFrameProcessor(
           return () => {
             'worklet'
             // dispose the Frame and the SkImage/Texture
-            image.dispose()
+            // image.dispose() // we do not dispose our mocked image :)
             nativeBuffer.delete()
           }
         }

Note

I'm doing some tricks to print the preview at 40% of its original size for better debugging However, at the end of the day, this code just simplifies to:

const srcRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
const dstRect = Skia.XYWHRect(0, 0, frameWidth, frameHeight);
canvas.drawImageRect(image, srcRect, dstRect, currentPaint);

portrait 0deg	landscape-left 90deg
portrait-upside-down 180deg	landscape-right 270deg

rodgomesc · 2024-04-30T03:02:34Z

Oh wait, this solves the misalignment of the frame inside the canvas; however, it causes another issue: the image buffer is now rotated 90 degrees clockwise inside a (0 deg portrait) preview for me. I think this is expected since we are lacking orientation support for Android, @mrousavy?

mrousavy · 2024-04-30T07:04:38Z

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

rodgomesc · 2024-04-30T10:04:05Z

I am applying the orientation inside the render(..) function, so actually it is expected that the Preview looks correct.

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android, and in render you are just doing getRotationDegrees(frame.orientation)
so i can't visualize the full picture here, of how did you got the image buffers always in portrait without a counter-clockwise rotation of -90deg in the current code that's running in render() {...}

react-native-vision-camera/package/android/src/main/java/com/mrousavy/camera/frameprocessors/Frame.java

Lines 89 to 93 in a53166b

 public Orientation getOrientation() throws FrameInvalidError { 

 assertIsValid(); 

 int degrees = imageProxy.getImageInfo().getRotationDegrees(); 

 return Orientation.Companion.fromRotationDegrees(degrees); 

 }

mrousavy · 2024-04-30T13:15:58Z

i mean, frame.orientation will always returns landscape-right since orientation support not being complete on android,

no, frame.orientation is an exception here - this is the sensor relative orientation of the ImageProxy, so this is correct as it is.

pweglik · 2024-05-08T08:07:02Z

Any news on this? I'm also experiencing this misplacement. I observed that it depends on format and video resolution:

Without passing format props to Camera:

Default format got from useCasmeraFormat():

  const format = useCameraFormat(device, []);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}

const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}

const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);

Log:

{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}

![2024-05-08 10 01 42]()

So in case of lower resolutions (default or set manually) only bottom part is cut off, while in higher resolutions entire frame is misplacement. Maybe that somehow helps you

OleksiiZdaly · 2024-05-14T08:51:09Z

Є новини з цього приводу? Я також відчуваю це неправильне розташування. Я помітив, що це залежить від формату та роздільної здатності відео:

Без передачі параметрів формату в камеру:

2. Формат за замовчуванням отримано з `useCasmeraFormat()`:
  const format = useCameraFormat(device, []);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 2160, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 3840}
const format = useCameraFormat(device, [
    { videoResolution: { width: 600, height: 400 } },
  ]);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 480, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 640}
const format = useCameraFormat(device, [
    { videoResolution: { width: 1200, height: 800 } },
  ]);
  console.log(format);
Журнал:
{"autoFocusSystem": "contrast-detection", "fieldOfView": 80.98175244802063, "maxFps": 30, "maxISO": 3200, "minFps": 7, "minISO": 100, "photoHeight": 2160, "photoWidth": 4096, "supportsDepthCapture": false, "supportsPhotoHdr": false, "supportsVideoHdr": false, "videoHeight": 720, "videoStabilizationModes": ["off", "cinematic"], "videoWidth": 1280}
Таким чином, у разі нижчої роздільної здатності (за замовчуванням або встановленої вручну) обрізається лише нижня частина, тоді як у вищій роздільній здатності зміщується весь кадр. Можливо, це якось тобі допоможе

For some reason, the format parameters don't change anything for me, the behavior is the same as yours in the second example. And the same problem with useSkiaFrameProcessor. On the emulator, the screen is completely black, and on real devices, part of it is. Another strange thing is that the frame itself processes frames rotated 90 degrees, I noticed this when my model started drawing rectangle around object on a black screen turned to the horizontal position. In IOS all ok.

mrousavy · 2024-05-21T15:48:44Z

Another strange thing is that the frame itself processes frames rotated 90 degrees

That's because of how Cameras work. Sensors are in 90deg orientation. See frame.orientation.

tomerh2001 · 2024-05-29T13:35:14Z

I also encountered this when using useSkiaFrameProcessor:

but useFrameProcessor works normally.

EnzoDomingues · 2024-06-02T17:47:07Z

I just changed this line

-              canvas.rotate(rotation, frame.width / 2, frame.height / 2)
+              canvas.rotate(rotation, frame.width / 2, frame.height / 1.5)

on useSkiaFrameProcessor.ts
Not it's working as expected

mrousavy · 2024-06-03T13:27:43Z

Created a fix for this here; #2931

Can you guys test if this works for you? :)

EnzoDomingues · 2024-06-03T15:16:24Z

Created a fix for this here; #2931

Can you guys test if this works for you? :)

@mrousavy
Hey, for me, it's even worse
I'm using Moto G84

EnzoDomingues · 2024-06-03T15:25:34Z

@mrousavy
I was using this format before

const format = useCameraFormat(device, [
    {
      videoResolution: {
        width: 480,
        height: Platform.OS === 'ios' ? 640 : 720,
      },
    },
  ]);

Using this format, even the colors are changing

rodgomesc · 2024-06-03T16:46:14Z

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

EnzoDomingues · 2024-06-03T16:49:22Z

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

mrousavy · 2024-06-03T18:36:01Z

Hm. Matrixes are really hard to get right.

rodgomesc · 2024-06-04T02:44:11Z

@EnzoDomingues can you leave the format, aspect ration configurations at the default or use them exactly like in this example app and send a new screenshot?

The first screenshot is default, without any format

yup, i can confirm, it's worse for me as well

rodgomesc · 2024-06-04T02:45:57Z

@EnzoDomingues can you confirm if applying this patch #2762 (comment) the frame keeps centered for you?

mrousavy · 2024-06-04T10:20:59Z

Try again now - I think it now works :)

EnzoDomingues · 2024-06-04T16:39:39Z

@mrousavy I added your changes locally and the video is not showing anymore

EnzoDomingues · 2024-06-04T17:06:53Z

@EnzoDomingues can you confirm if applying this patch #2762 (comment) the frame keeps centered for you?

@rodgomesc, with your patch

mrousavy · 2024-06-04T17:33:50Z

@EnzoDomingues can you log frame.orientation?

EnzoDomingues · 2024-06-04T17:38:39Z

@mrousavy sure

 LOG  landscape-right
 LOG  landscape-right
 LOG  landscape-right

mrousavy · 2024-06-04T17:40:33Z

Ah that's why! Can you try to adjust this code here:

react-native-vision-camera/package/src/skia/useSkiaFrameProcessor.ts

Lines 75 to 78 in cd18368

 case 'landscape-right': 

 // rotate two flips on (0,0) origin and move X + Y into view again 

 canvas.translate(frame.height, frame.width) 

 canvas.translate(270, 0)

I don't have a phone where it's sensor orientation is landscape-right, so I cannot test this myself. Maybe just switch out the translations, use 0 or whatever - if you find a fix pls let me know.

mrousavy · 2024-06-04T17:41:26Z

oh lol after posting this I just realized I used the wrong function.... lmao see 38c6983

now that should work

EnzoDomingues · 2024-06-04T17:43:11Z

@mrousavy

   // rotate two flips on (0,0) origin and move X + Y into view again
        canvas.translate(0, frame.width)
        canvas.rotate(270, 0, 0)

This works for me instead canvas.translate(frame.height, frame.width)
Could you update it?

mrousavy · 2024-06-04T18:03:24Z

aha interesting... can you send a PR?

EnzoDomingues · 2024-06-04T18:41:26Z

@mrousavy MR created :)

tomerh2001 · 2024-06-05T08:01:38Z

Is there a release containing this fix?

mrousavy · 2024-06-05T08:32:05Z

Not yet.

tomerh2001 · 2024-06-05T09:21:08Z

Not yet.

Waiting on this for prod release

karventhan13 added the 🐛 bug Something isn't working label Apr 20, 2024

This comment was marked as spam.

Sign in to view

mrousavy changed the title ~~🐛 [^4.0.0-beta.16] - Camera is not aligned properly when using useSkiaFrameProcessor~~ 🐛 Camera is not aligned properly when using useSkiaFrameProcessor Apr 22, 2024

mrousavy mentioned this issue Jun 3, 2024

fix: Fix Skia Frame Processor transformation matrix #2931

Merged

mrousavy closed this as completed in #2931 Jun 4, 2024

EnzoDomingues mentioned this issue Jun 4, 2024

fix: Fixed canvas.translate on withRotatedFrame for landscape-right #2939

Merged

🐛 Camera is not aligned properly when using useSkiaFrameProcessor #2762

🐛 Camera is not aligned properly when using useSkiaFrameProcessor #2762

Comments

karventhan13 commented Apr 20, 2024

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

This comment was marked as spam.

mrousavy commented Apr 21, 2024

mrousavy commented Apr 21, 2024

Space6188 commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

mrousavy commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

rodgomesc commented Apr 22, 2024

mrousavy commented Apr 22, 2024

Space6188 commented Apr 22, 2024

rodgomesc commented Apr 29, 2024

AndyRC6 commented Apr 29, 2024 • edited

rodgomesc commented Apr 30, 2024 • edited

rodgomesc commented Apr 30, 2024 • edited

mrousavy commented Apr 30, 2024

rodgomesc commented Apr 30, 2024 • edited

mrousavy commented Apr 30, 2024

pweglik commented May 8, 2024

OleksiiZdaly commented May 14, 2024

mrousavy commented May 21, 2024

tomerh2001 commented May 29, 2024 • edited

EnzoDomingues commented Jun 2, 2024 • edited

mrousavy commented Jun 3, 2024

EnzoDomingues commented Jun 3, 2024 • edited

EnzoDomingues commented Jun 3, 2024 • edited

rodgomesc commented Jun 3, 2024

EnzoDomingues commented Jun 3, 2024

mrousavy commented Jun 3, 2024

rodgomesc commented Jun 4, 2024

rodgomesc commented Jun 4, 2024

mrousavy commented Jun 4, 2024

EnzoDomingues commented Jun 4, 2024

EnzoDomingues commented Jun 4, 2024

mrousavy commented Jun 4, 2024

EnzoDomingues commented Jun 4, 2024

mrousavy commented Jun 4, 2024

mrousavy commented Jun 4, 2024

EnzoDomingues commented Jun 4, 2024 • edited

mrousavy commented Jun 4, 2024

EnzoDomingues commented Jun 4, 2024

tomerh2001 commented Jun 5, 2024

mrousavy commented Jun 5, 2024

tomerh2001 commented Jun 5, 2024

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

🐛 Camera is not aligned properly when using `useSkiaFrameProcessor` #2762

AndyRC6 commented Apr 29, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

rodgomesc commented Apr 30, 2024 •

edited

tomerh2001 commented May 29, 2024 •

edited

EnzoDomingues commented Jun 2, 2024 •

edited

EnzoDomingues commented Jun 3, 2024 •

edited

EnzoDomingues commented Jun 3, 2024 •

edited

EnzoDomingues commented Jun 4, 2024 •

edited