WIP: Dev openglaccelerated playback

[nolyn]

A while back I worked on fast playback using OpenGL, I now made some effort to merge this into YUView. This is a basic working version, but still needs a lot of refinement.

How to use:

• Open YUView, press Ctrl + G, open a YUV file (420, 8/10 bit), then press play.
• Alternatively open a compressed video, again only 420, 8/10 bit supported so far.
• On my laptop (i7, no GPU) i can fluently playback BasketballDrive (1920x1080 50fps 8bit 420), before it was around 20fps (without caching).

Issues

• After loading the first sequence, FileDialog for opening others is not always displayed (though still works in the background, arrow keys, and enter work). Happens more often when playback is paused. If unpaused dialog opens, possibly with some delay.
• The way that the data transfer to the opengl view is implemented should likely be changed. UpdateFrame will always also signal format to opengl widget, might be simplified somehow
• At the moment there is only support for YUV 420, 8 and 10 bit. More formats can be implemented by extending the shader. At the moment bt709 is hard coded.
• Only has been tested under Linux. There might be issues with Windows: https://doc.qt.io/qt-5/windows-issues.html
• Requires OpenGL 3.3, I think this is a widely supported version. No error handling has been implemented. However, there is some debugging functionality, which can be enabled by adding "CONFIG += console" to the project files. An alternative to OpenGL could be using Vulkan.
• There is no switching to classic YUView playback implemented yet, i simply commented the code doing the internal YUV2RGB conversion.

Todo

• Test on device without opengl support, implement message/popup/etc. to inform about this case
• Using pbo should further increase performance

[nolyn]

@ChristianFeldmann what do you think about this? Nice to have in YUView? Does it work in Windows? I think the first two/main issues above might be connected. How to best move the data around in YUView?

[ChristianFeldmann]

Hi Johannes,
I saw you working on this (got some notifications about pushes to the repo). I think this would definitely be cool to have another render mode if it really works much faster to do the conversion. However, we would have to make sure that it always works for the features that we have (or switch to the SW rendering if it does not).
I will test it on windows. I also have a mac so I can test that too.

[nolyn]

I think the speedup is considerable, I already mentioned HD above. For higher resolutions my laptop hits its limit, but 8K plays still 4 times faster than before (without cache, as if my RAM has enough cache for 8K anyway xD). I think the bottleneck will be how fast the data can be read from disk and moved to the GPU. Still have to test it on one of the desktops with GPU in the institute, though I'm not sure they have SSD drives.
I was thinking this could be a fast playout mode that can be switched to / enabled optionally. Not sure it makes sense to integrate it with everything, for example statistics are viewed while paused anyway. I also have to confess that I don't know what all the things are that YUView can do now, I only use it for YUV files. Would be nice to integrate it with video decoders though.

[aachenmax]

@nolyn Sweet!

[nolyn]

@aachenmax did you try it? does it work on windows?

[ChristianFeldmann]

I just tested it and you seem to be using https://github.com/g-truc/glm which I must compile first. I will take a look how big it is what we need. Its MIT license so we could theoretically also just get the code into our codebase...

[nolyn]

Probably glm can be removed completely. Only using some of its data types. Not any functions (anymore). I already use a QMatrix4x4 as well, I'll check wheter I can just replace the glm types with qt types.

[nolyn]

I just tested it and you seem to be using https://github.com/g-truc/glm which I must compile first. I will take a look how big it is what we need. Its MIT license so we could theoretically also just get the code into our codebase...

the dependency should be gone now

[ChristianFeldmann]

Nice I can confirm that it builds now on Windows. But unfortunately I don't see anything when opening a sequence. Are there any restrictions that I have to be aware of? I opened BQ-Terrace (1080p, 4:2:0).

[nolyn]

420, 8 and 10 bit should work. both YUV or coded files. it uses a separate window though, did you press Ctrl+G?

[ChristianFeldmann]

Ah yes now it shows the GL View with the sequence!
But I am only able to get like 30 fps on my machine. And some proper scaling artifacts 🙂

[nolyn]

I'll see if I can borrow a windows machine to do some testing. I suspect the 30 fps will be due to the frame rate that windows uses for the monitor

[nolyn]

reicht zum bauen unter windows der qt installer: https://www.qt.io/offline-installers? oder was brauch ich dazu noch nen extra compiler ...

[nolyn]

i'm trying to build on windows with mingw64, which came along with the Qt installer. However it fails with
No rule to make target YUViewLib.lib required by YUView.exe
YUViewApp and YUViewUnitTest build. I can also open the YUViewLib.pro project files separately and build it without issue.

[nolyn]

i also tried msvc 2017, but without success
cl -c -nologo -Zc:wchar_t -FS -Zc:rvalueCast -Zc:inline -Zc:strictStrings -Zc:throwingNew -Zc:referenceBinding -Zc:__cplusplus -Zi -MDd -W3 -w34100 -w34189 -w44996 -w44456 -w44457 -w44458 -wd4577 -wd4467 -EHsc /Fd./YUViewLib.pdb -DUNICODE -D_UNICODE -DWIN32 -D_ENABLE_EXTENDED_ALIGNED_STORAGE -DWIN64 -DNOMINMAX -DYUVIEW_HASH=\"0\" -DYUVIEW_VERSION=\"0\" -DQT_QML_DEBUG -DQT_OPENGL_LIB -DQT_WIDGETS_LIB -DQT_GUI_LIB -DQT_XML_LIB -DQT_CONCURRENT_LIB -DQT_NETWORK_LIB -DQT_CORE_LIB -I..\..\YUView\YUViewLib -I. -I..\..\YUView\YUViewLib\src -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtOpenGL -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtWidgets -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtGui -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtANGLE -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtXml -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtConcurrent -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtNetwork -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\include\QtCore -I. -I. -I/include -IC:\Qt\Qt5.12.9\5.12.9\msvc2017_64\mkspecs\win32-msvc -Fo @C:\Users\xx\AppData\Local\Temp\moc_mainwindow.obj.7488.76219.jom moc_mainwindow.cpp lib /NOLOGO /OUT:YUViewLib.lib @C:\Users\xx\AppData\Local\Temp\YUViewLib.lib.7488.80969.jom cd YUViewApp\ && ( if not exist Makefile C:\Qt\Qt5.12.9\5.12.9\msvc2017_64\bin\qmake.exe -o Makefile C:\Users\xx\Desktop\YUView\YUViewApp\YUViewApp.pro -spec win32-msvc "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:\Qt\Qt5.12.9\Tools\QtCreator\bin\jom.exe -f Makefile Error: dependent '\YUViewLib\YUViewLib.lib' does not exist. jom: C:\Users\xx\Desktop\build-YUView-Desktop_Qt_5_12_9_MSVC2017_64bit-Debug\Makefile [sub-YUViewApp-make_first] Error 2 23:28:47: Der Prozess "C:\Qt\Qt5.12.9\Tools\QtCreator\bin\jom.exe" wurde mit dem Rückgabewert 2 beendet. Fehler beim Erstellen/Deployment des Projekts YUView (Kit: Desktop Qt 5.12.9 MSVC2017 64bit)

@ChristianFeldmann, @aachenmax any ideas?

[ChristianFeldmann]

😆 oha.
So I think Mingw was not tried in a long long time. Also in CI we compile everything using the visual Cpp compiler. Here is also some info how we do it: https://github.com/IENT/YUView/wiki/Compile-YUView
And of course The CI files show you the recipies that Github actions uses.
With qmake you can create visual studio solution files. Or I have the C++ build environment (not Visual Studio) installed and then I just use qmake and nmake.

[nolyn]

i can build now :). at some point while setting up the laptop i borrowed (since it has windows) i deleted .qmake.conf. had to also uninstall a few old build chains before i got anywhere and then i didn't realize that file was missing anymore.

however I also managed to build without that file by using relative paths in the project files, see the new branch develop_remove_top_soure_and_builddir_variables. if that also works for the other build environments it is probably a matter of taste how to do it.

[nolyn]

on windows i see a relation between the window size and the fps. only tested BasketballDrive. with a small window it reaches 50fps, but it goes down to about 30 when maximized. i'll also have to test it with a linux livesystem on that laptop so i have an actual comparison on the same device.
i also had to create a ramdisk where i put the video, without i got only 20fps.

[nolyn]

Hi, any updates? Do you want me to test this on nvidia geforce since it supports it now (geforce windowed OpenGl 10 bit mode was activated)?

sorry, I don't have time to work on this anymore (and also no device that could do 10 bit output). Also it is now very much out of date compared to current YUView version.

I think you could still use it though to get started if you want to implement opengl windows output for 10bit. Just getting opgengl output window to work properly in qt took quite some effort and testing. It would also require quite some internal changes in YUView to get good performance. YUView does conversion for output to RGB internally on CPU. This could be done instead in the GPU shader. But it will take quite some implementation work.

@ChristianFeldmann maybe just close this? and keep in mind its there, if anyone wants to implement this in the future.

[ChristianFeldmann]

Hi.
I converted this to a draft PR. I looked into the code and I think its a really good idea to make openGL the new default drawing method for YUView. We could do all color conversions from YUV to RGB in shaders which would be a great speedup in general (maybe except for Mac where openGL is not the best performing way of drawing stuff). However, I also currently don't have much time to really push this forward and completely make this work on all platforms.
I hope I will soon though.