I don't really understand how you're supposed to control a UI with an instrument.
What do you do on a PS3 with your instrument in these cases?

The actual ps3 buttons on them

The guitar in the video also has PS3 buttons. I've heard similarly that some instruments mapped through MMJoystick have issues interacting with the Native Interface enabled but I haven't been able to replicate it so will ask around for anyone else with this issue.

The instruments themselves also show up as HID devices to the PS3 rather than an actual midi instrument. All of the MIDI code that I wrote for RPCS3 does this conversion that is normally done by a separate hardware device with gamepad buttons it.

In case you are curious, the Wii version of this device looks like this but this one has been converted to work on PS3 instead. (And of course there exists an actual PS3 version of it.)

For RPCS3, we instead emulate the gamepad buttons with additional midi notes that would have otherwise been ignored by this device.

Please try with #15180

And also try to open the home menu with the ps button on the instrument.

Tested with what was just pushed an hour ago
#15180 (comment)

Native interface still does not respond to a passthrough Keyboard
RPCS3.log
https://www.youtube.com/watch?v=54bit_D4E0M

This was marked as closed but the problem persists on USB Passthrough instruments.
Tested with a PS3 Rock Band 3 Mustang Controller (VID=0x12ba, PID=0x2430) running with passthrough (Zadig).
The game works fine but it locks up when the "Invite Friends" menu pops up and the PlayStation guide button does nothing.
Rock Band 3 [BLUS30463].log.gz

Does it work on PS3?
The game is supposed to call cellPadLddDataInsert, which it doesn't.

it's set to trace. Can you put it to notice and create a new log?

It does work on PS3. The PlayStation button would bring up the respective menu, the PSN invite menu flow would work via the gamepad buttons, etc.
Regarding trace for log, would this be on the right click menu for RPCS3's log or somewhere else?
Also could this problem be caused by emulation pausing when the RPCN friend list menu is up?

I think you have to manually edit the config.yml and set
Log: {}
to
Log: {cellPad: Trace}

Hopefully I did it right.
RPCS3.log.gz

ah... it's
Log: {sys_io: Trace}

Here it is.
RPCS3.log.gz

I added some more logging to this one:
rpcs3.zip

Getting this error on starting that RPCS3 exe.

rpcs3.exe - Entry Point Not Found

The procedure entry point ?fromString@QDateTime@@sa?AV1@AEBVQString@@0h@Z could not be located in the dynamic link library C:\Games\RPCS3\rpcs3.exe.

OK

uhm, right, I'm already on a newer Qt version.
You could try with the dlls and qt6 folder from #15473.

Here it is.
RPCS3.log.gz

Based on the log, it seems to call cellPadLddDataInsert periodically, but doesn't set the length field, so it never really sets any data.
Also, it doesn't seem to do anything while the dialog is open.
I've made another one that logs the buttons even if the length is 0.
rpcs3.zip

Same results.
New log:
RPCS3.log.gz

Did you press up/down/cross on the guitar a couple of times?

I've changed a couple of things.
I doubt that the friend list works, but maybe the home menu does now.

rpcs3.zip

Did you press up/down/cross on the guitar a couple of times?

Yep!
RPCS3.log.gz

Just as you said, home menu and interacting with it works. When the menu is pulled up, the emulation persists in the background like on real hardware. The RPCN friend menu seems to pause emulation entirely while it's up and it does not respond until dismissed by using an alternative input method (like a keyboard) is used. On hardware, the game behaves like how the Home Menu behaves at the moment.

Maybe open a separate issue with this specific dialog in this game.
I think we've proven that it works for all other dialogs.
I'll open a PR with the home menu changes for this one.