Draft: Add model dragging in XR headsets by controllers or hands

[FluorescentHallucinogen]

This pull request fixes a problem (described by me a year ago in the #3969 (reply in thread)) that users can't move and rotate models in the Meta Quest headsets.

[elalish]

Thanks, this is exciting! Now I just need to hunt down a Quest so I can test this personally...

[elalish]

Can you try to replace these @ts-ignores by casting, e.g. const group = this.presentedScene!.model as Group; or something? That's a lot more specific and helps me to understand the problem.

[elalish]

The material might already have an emissive factor - can you save this in userData and set back to the original instead?

[FluorescentHallucinogen]

This is to highlight (brighten) the 3D model when the controller ray intersects it. Please check out the video again. ;) This highlight is removed back in the cleanIntersected() method by calling the code.object.material.emissive.setHex(0x000000).

[FluorescentHallucinogen]

Ah, I think I get it, you mean that the 3D model can have the emissive property set to a value other than 0x000000 before the controller ray intersects the 3D model? Then a question: can there be more than one 3D model in a scene at the same time?

BTW, highlighting (brightening) the 3D model on the hover is an optional improvement. I mean, we can opt out of that and simplify the implementation.

[elalish]

Exactly, but this caused me to notice a potentially bigger issue. We have one 3D model, but it can be composed of an arbitrary hierarchy of objects. We don't expose those sub-meshes in our public API, so they probably should not be independently selectable. Instead we should move everything when anything is selected. And we already have a highlighting concept in WebXR: the PlacementBox - maybe best to just use that.

I'm interested in what UX you're going for here - it might be nice to make it as similar to our existing AR UX as possible, given controllers. I'm open to other ideas, but I'd like to know how you're envisioning the experience.

[FluorescentHallucinogen]

Could you please share links to an example where there is more than one 3D model in the one scene at the same time, and an example where one complex 3D model consists of multiple 3D models (parts)?

[FluorescentHallucinogen]

As you can see in the video, a little highlighting (brightening) of the 3D model on the hover is not really needed at all. :)

[elalish]

I like your style 😄. Fair point, though it makes a bit less sense with furniture. I need to think about this. Will this interaction work the same way with hands? Will we only support moving objects within arm's reach? I wish we had a UX person dedicated to thinking these use cases through, but I'm afraid it's just us for now.

[FluorescentHallucinogen]

Will this interaction work the same way with hands?

Yes, hands are also controllers in WebXR. Moreover, on Meta Quest you can switch between controllers and hands within the same XR session. You can put the controllers down and after a few seconds the controller will go to sleep and there will be an automatic switch to the hands. If you pick up the controller, it will wake up and switch back.

Will we only support moving objects within arm's reach?

No, this approach is already essentially standard, it works with both small and large objects, both close and distant objects (the ray is long enough).

I'll try to record other videos for you to make it clearer. But it would be better if you try it yourself (use different apps and games for a few days) when you get the headset. That will clear up most of the questions. :)

I wish we had a UX person dedicated to thinking these use cases through, but I'm afraid it's just us for now.

It's been already researched. Check out e.g. the Mixed Reality Design Language by Microsoft. ;)

Please also note that currently the user has no way to interact with the 3D model in the XR headset at all.

So my goal in this pull request is to add the simplest basic implementation. This will already improve the user experience significantly!

Then we can improve the implementation. Let's move step by step. ;)

[FluorescentHallucinogen]

See https://learn.microsoft.com/en-us/windows/mixed-reality/design/point-and-commit. 😉

[elalish]

Thanks a lot, that's quite helpful!

[elalish]

Are these new Three.js APIs? Otherwise we should have types for them.

[FluorescentHallucinogen]

Ok, I'll try adding types everywhere again.

[elalish]

Regarding controllers of the wrong size, you can see further down this file that scene.scale is changed when the object is scaled by pinch on a phone. We may need to add another level of object hierarchy to the AR scene to fix this. However, I believe that scale starts at 1, so if you're seeing a problem from the beginning, it may be a different issue. Can you add a screen recording by chance?

[elalish]

Okay, I just saw your video on the other thread, thanks. Interesting, that does seem like a different problem than scene scale. Position maybe? Let's move discussion from the thread to this PR, FYI @cabanier.

[FluorescentHallucinogen]

Yes, it's a position problem, not a scale problem. The controllers seems small because they are far away from the user. E.g. I've found that initially the controller model is in the right place, but then the 3D model drops to the floor and the controller gets lower.

Changing the isFirstView variable to false:

// WebXR may return multiple views, i.e. for headset AR. This
// isn't really supported at this point, but make a best-effort
// attempt to render other views also, using the first view
// as the main viewpoint.
let isFirstView: boolean = true;
for (const view of pose.views) {
  this.updateView(view);

  if (isFirstView) {
    this.moveToFloor(frame);

    this.processInput(frame);

    const delta = time - this.lastTick!;
    this.moveScene(delta);
    this.renderer.preRender(scene, time, delta);
    this.lastTick = time;

    scene.renderShadow(this.threeRenderer);
  }

  this.threeRenderer.render(scene, scene.getCamera());
  isFirstView = false;
}

and commenting/removing the following code:

const {theta, radius} =
    (element as ModelViewerElementBase & ControlsInterface)
        .getCameraOrbit();
// Orient model to match the 3D camera view
const cameraDirection = xrCamera.getWorldDirection(vector3);
scene.yaw = Math.atan2(-cameraDirection.x, -cameraDirection.z) - theta;
this.goalYaw = scene.yaw;

position.copy(xrCamera.position)
    .add(cameraDirection.multiplyScalar(radius));

this.updateTarget();
const target = scene.getTarget();
position.add(target).sub(this.oldTarget);

this.goalPosition.copy(position);

fixes the problem.

[elalish]

Do you understand why this is? Any recommendation for a better approach?

[FluorescentHallucinogen]

Controllers are added to the scene. The position of the scene changes, the controllers move with it.

[MickGerrit]

@FluorescentHallucinogen I tried out the demo. Very cool! The fixed version is already a big improvement for viewing on a Meta Quest.

[sprengerst]

@FluorescentHallucinogen 👍 great job so far.

[sprengerst]

I tried your solution with other models, the "drag and drop" doesn't seem to work for all GLBs.

[sprengerst]

return raycaster.intersectObjects(group.children, **true**);

This should be set to true

[FluorescentHallucinogen]

@sprengerst But why? From what I can see, it's true by default. See https://threejs.org/docs/#api/en/core/Raycaster.intersectObjects.

[FluorescentHallucinogen]

@elalish What are the chances of you getting your hands on Meta Quest? I thought it was a must have for the model-viewer maintainer. :)

[elalish]

Well, now that we're looking to get serious about adding headset support, I'd agree. I'm going to get one ordered now.

[FluorescentHallucinogen]

we're looking to get serious about adding headset support

That's great news!

I'm going to get one ordered now.

Please let me know once you order the headset and which model (Meta Quest 3 or Meta Quest Pro). And when you get it in your hands. ;)

I expect when we add support for XR headsets and test the code on Meta Quest, we will automatically have Apple Vision Pro support since it's the same WebXR standard here and there. :)