MV3 user's mic and cam permissions using iframe #821

esphoenixc · 2023-01-25T04:15:02Z

I am building a chrome extension that records user's cam, mic, and screen.

For the screen, I think I took care of it by using desktopCapture API in background.js.

The problem comes in for user's cam and mic.

I have an iframe injected into the shadow dom. From that iframe, I request for the permissions of the use of the user's cam and mic by using "navigator.mediaDevices.getUserMedia({audio: true, video:true})". The reason for requesting mic and cam permissions inside the iframe is because I want to ask for the permissions on behalf of my extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your camera and microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

I want to display user's cam and mic streams in content script, not inside the iframe. In order for me to stream the user's cam and mic in content script, somehow I need to find a way to get the streams of the cam and mic from the iframe and send it over to content script and display the streams. I believe this is where chrome.tabCapture API comes in.

However, in manifest version 3, I cannot use chrome.tabCapture in backgrund.js (now also known as service workser). I think that's causing a major issue in this case.

As I researched more into this specific case, I've found that there's a new feature called, offscreen API.

Can I use offscreen API to achieve what I'm trying to achieve?

requesting permission from the permission.html inside the iframe, which will be created using offscreen API
use that permission in content script to use navigator.mediaDevices.getUserMedia() to get the streams of the user's cam and mic.

Is this even possible in manifest version 3 currently?

if what I wrote or am trying to accomplish is unclear, please let me know.

Here are the references I've gone over to solve this issue :
https://bugs.chromium.org/p/chromium/issues/detail?id=1214847
https://bugs.chromium.org/p/chromium/issues/detail?id=1339382
https://stackoverflow.com/questions/74773408/chrome-tabcapture-unavailable-in-mv3s-new-offscreen-api
https://stackoverflow.com/questions/66217882/properly-using-chrome-tabcapture-in-a-manifest-v3-extension
#627
w3c/webextensions#170
https://groups.google.com/a/chromium.org/g/chromium-extensions/c/ffI0iNd79oo/m/Dnoj6ZIoBQAJ?utm_medium=email&utm_source=footer

The text was updated successfully, but these errors were encountered:

patrickkettner · 2023-03-03T17:35:15Z

Hi @esphoenixc!
I believe offscreenDocument should allow you to accomplish this? Let us know if you hit any issues with it.

I am tagging this as a potential sample the team could put together

M-SAI-SOORYA · 2023-03-28T17:33:27Z

Yes, it is possible to use the Offscreen API to achieve what you are trying to do in Manifest version 3.

To start, you can use the Offscreen API to create a hidden canvas element in your content script. You can then use navigator.mediaDevices.getUserMedia() to get the user's cam and mic streams inside your iframe and pass them to your background script.

From your background script, you can then use the chrome.runtime.sendMessage() method to send the streams to your content script, where you can display them on the hidden canvas element using CanvasRenderingContext2D.drawImage()

KiranNadig62 · 2023-08-09T10:24:44Z

What's the status of the sample? We are unable to get this working. Offscreen APIs cannot obtain media permissions from our testing.

jpmedley · 2023-08-10T16:40:37Z

I'm a little rusty on the issue, but I think you might be looking for this: https://developer.chrome.com/docs/extensions/mv3/screen_capture/

If so, I apologize for forgetting that we had an open ticket on this.

aakash232 · 2023-08-11T06:34:57Z

I'm a little rusty on the issue, but I think you might be looking for this: https://developer.chrome.com/docs/extensions/mv3/screen_capture/

If so, I apologize for forgetting that we had an open ticket on this.

@jpmedley The above solution will be wrt some specific tab. (Getting stream from tab)
Main goal is as stated in the issue :-

The reason for requesting mic and cam permissions inside the iframe is because I want to ask for the permissions on behalf of my extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your camera and microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

How do we get access to Media from the user's microphone using the offscreen API?
I am talking about something like this.

(Inside offscreen.js)

function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((response) => {
        if (response.id) 
            resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission:", error);
        if (error.message === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        }
        reject(error);
      });
  });
}

Currently, this throws a NotAllowedError: Failed due to shutdown

aakash232 · 2023-08-16T09:49:32Z

@esphoenixc

Found this alternative for your requirement. (Without offscreen API)

Attaching steps for reference if anyone needs a possible workaround until offscreen api stuff is clear.

This will ask for the permissions on behalf of extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

In your Content Script (say, `injectDOM.js`)

Create and inject an iframe to the page where extension will be used.

const iframe = document.createElement("iframe");
iframe.setAttribute("hidden", "hidden");
iframe.setAttribute("id", "permissionsIFrame");
iframe.setAttribute("allow", "microphone");
iframe.src = chrome.runtime.getURL("requestPermissions.html");
document.body.appendChild(iframe);

This will have the src file requestPermissions.html. The script file for this page, requestPermissions.js will have the code to request permission.

requestPermissions.html

<!DOCTYPE html>
<html>
  <head>
    <title>Request Permissions</title>
    <script>
      "requestPermissions.js";
    </script>
  </head>
  <body>
    <!-- Display loading or informative message here -->
  </body>
</html>

requestPermissions.js

/**
 * Requests user permission for microphone access.
 * @returns {Promise<void>} A Promise that resolves when permission is granted or rejects with an error.
 */
async function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    // Using navigator.mediaDevices.getUserMedia to request microphone access
    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((stream) => {
        // Permission granted, handle the stream if needed
        console.log("Microphone access granted");
        resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission", error);

        // Handling different error scenarios
        if (error.name === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        } else {
          reject(error);
        }
      });
  });
}

// Call the function to request microphone permission
getUserPermission();

In `manifest.json`

"web_accessible_resources": [
    {
      "resources": ["requestPermissions.html", "requestPermissions.js"],
      "matches": ["<all_urls>"]
    }
],

juxnpxblo · 2023-09-27T14:32:46Z

I figured that you're able to get microphone stream on offscreen if user had previously consented to microphone permission for your extension, however, an offscreen document can't ask permission itself, so you must get it from some other context

on my demo using an action popup, I setup a popup.html with a script tag sourcing from popup.js, and in popup.js I had a getUserMedia({ audio: true }) call, which triggered the permission prompt " wants to use your microphone" on first time

after I consented, I was able to call getUserMedia({ audio: true }) from my offscreen document and successfully get a microphone stream; before consenting, I would get a DOMException: Failed due to shutdown

aakash232 · 2023-09-29T04:21:22Z

I figured that you're able to get microphone stream on offscreen if user had previously consented to microphone permission for your extension, however, an offscreen document can't ask permission itself, so you must get it from some other context

on my demo using an action popup, I setup a popup.html with a script tag sourcing from popup.js, and in popup.js I had a getUserMedia({ audio: true }) call, which triggered the permission prompt " wants to use your microphone" on first time

after I consented, I was able to call getUserMedia({ audio: true }) from my offscreen document and successfully get a microphone stream; before consenting, I would get a DOMException: Failed due to shutdown

True. Post consent, we can use getUserMedia({ audio: true }) in offscreen API. That's what I am using currectly.

Only thing was, your way of asking for consent (via popup.js) triggers "www.google.com/*** wants to use your microphone" instead of "[myExtension] wants to Use". Correct me if I am wrong.

Thanks

athioune · 2023-10-01T15:33:21Z

Hi @aakash232,

Do you happen to have a full example of your solution ? (Because with MV3 I always have a not allowed error)

Thank you !

aakash232 · 2023-10-01T17:29:01Z

Hi @aakash232,

Do you happen to have a full example of your solution ? (Because with MV3 I always have a not allowed error)

Thank you !

@athioune
I will give an example scenario from my side. Let me know if you needed this or something else.

OBJECTIVE

GOAL : Get mic permissions wrt extension's context and record audio via offscreen API

ISSUE : Getting not allowed error if we request mic permissions inside offscreen doc

SOLUTION : Get permissions via external iframe, once consent is received handle audio capture and other tasks inside offscreen doc.

STEPS

Step 1 : To get permissions, follow my previous reply steps

Step 2 : Handle recording. (All the below steps can be achieved via offscreen doc)

Create offscreen doc initially

  await chrome.offscreen
    .createDocument({
      url: offscreen.html,
      reasons: [USER_MEDIA],
      justification: "keep service worker running and record audio",
    })

Inside offscreen.js

Check status of mic permissions

      if (result.state === "granted") {
        // we have the permissions, continue next steps
        handleRecording();
      } else if (result.state === "prompt") {
        // prompt user to get the permissions
        // Call the function in requestPermissions.js to request microphone permission [ getUserPermission() ]       
      } else if (result.state === "denied") {
        // permissions denied by user
      }
    });

In handleRecording() you can get audio input devices.

navigator.mediaDevices
      .enumerateDevices()
      .then((devices) => {
        // Filter the devices to include only audio input devices
        const audioInputDevices = devices.filter(
          (device) => device.kind === "audioinput"
        );
       // Use them accordingly
      })

Once we have the input devices, then we can capture audio from the device ID you wish to

const deviceId = audioInputDevices[0].deviceId;

    navigator.mediaDevices
      .getUserMedia({
        audio: {
          deviceId: { exact: deviceId },
        },
      })
      .then((audioStream) => {
           //get audio stream from your selected input device

         //pass the same for new media recorder instance 
          mediaRecorder = new MediaRecorder(audioStream);
        
          mediaRecorder.ondataavailable = (event) => {        
              chunks = [event.data];
             //handle chunks as needed            
          };

          mediaRecorder.onstop = handleStopRecordingFunction;
         
          //set time slice to capture data every X milliseconds
          mediaRecorder.start(5000);     
      });

athioune · 2023-10-02T09:21:56Z

Thank you very much @aakash232 it worked !

ddematheu · 2024-04-16T07:21:25Z

Sorry to revive this, but when I try to trigger permissions flow from the popup, I see an error: "Permissions Dismissed". The pop up is not even coming up. Anyone seen this?

aakash232 · 2024-04-17T07:41:30Z

Sorry to revive this, but when I try to trigger permissions flow from the popup, I see an error: "Permissions Dismissed". The pop up is not even coming up. Anyone seen this?

Received this error along with DOMException: Failed due to shutdown somewhere when I was trying to trigger the flow from the offscreen doc. To bypass this error, I went through the additional efforts of injecting iframe.

Few follow-up(s),

Did you check if the permissions aren't explicitly blocked by chrome for the extensions? (in manage extensions page).
The offscreen sample PR : Did the flow here helped you out?
If nothing works, can you please elaborate the flow via which you are showing the popup? (Is there something you are trying differently apart from the sample PR. Maybe more context can help in debugging).

esphoenixc changed the title ~~MV3 user's mic ans cam permissions using iframe~~ MV3 user's mic and cam permissions using iframe Jan 25, 2023

patrickkettner added the new sample label Mar 3, 2023

aakash232 linked a pull request Dec 20, 2023 that will close this issue

Add offscreen user media sample #1056

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MV3 user's mic and cam permissions using iframe #821

MV3 user's mic and cam permissions using iframe #821

esphoenixc commented Jan 25, 2023 •

edited

patrickkettner commented Mar 3, 2023

M-SAI-SOORYA commented Mar 28, 2023

KiranNadig62 commented Aug 9, 2023

jpmedley commented Aug 10, 2023

aakash232 commented Aug 11, 2023 •

edited

aakash232 commented Aug 16, 2023 •

edited

juxnpxblo commented Sep 27, 2023

aakash232 commented Sep 29, 2023

athioune commented Oct 1, 2023

aakash232 commented Oct 1, 2023

athioune commented Oct 2, 2023

ddematheu commented Apr 16, 2024

aakash232 commented Apr 17, 2024

MV3 user's mic and cam permissions using iframe #821

MV3 user's mic and cam permissions using iframe #821

Comments

esphoenixc commented Jan 25, 2023 • edited

patrickkettner commented Mar 3, 2023

M-SAI-SOORYA commented Mar 28, 2023

KiranNadig62 commented Aug 9, 2023

jpmedley commented Aug 10, 2023

aakash232 commented Aug 11, 2023 • edited

aakash232 commented Aug 16, 2023 • edited

Found this alternative for your requirement. (Without offscreen API)

In your Content Script (say, injectDOM.js)

requestPermissions.html

requestPermissions.js

In manifest.json

juxnpxblo commented Sep 27, 2023

aakash232 commented Sep 29, 2023

athioune commented Oct 1, 2023

aakash232 commented Oct 1, 2023

OBJECTIVE

STEPS

Inside offscreen.js

athioune commented Oct 2, 2023

ddematheu commented Apr 16, 2024

aakash232 commented Apr 17, 2024

esphoenixc commented Jan 25, 2023 •

edited

aakash232 commented Aug 11, 2023 •

edited

aakash232 commented Aug 16, 2023 •

edited

In your Content Script (say, `injectDOM.js`)

In `manifest.json`