cairosink and gpu buffer sharing (gstreamer conference 2013)
TRANSCRIPT
ongoing experiment, I may not have the best answers yet
Work sponsored by Samsung Research Americahttp://www.sisa.samsung.com/
I. Technologies, past and present
Cairo: 2D graphicsCairogles: 2D graphics in OpenGL(ES)
support for EGL/GLX/WGL
Sharing video buffers: before
- Add a specific cap (e.g. video/x-raw-gl) - subclass GstBuffer
Sharing video buffers: nowadays
- GstContext - GstVideoGLTextureUploadMeta - GstAllocator and GstMemory
- GstContext, system to share a GL context and more between elements - GstVideoGLTextureUploadMeta: a way to tell how to upload a buffer to GPU - GstAllocator and GstMemory: cleaner way to handle non-CPU memory
Sharing video buffers: nowadays
"modern" gl(es) sink implementationsknown to the author: - eglglessink (and libgstegl) - glimagesink
II. Integrating cairogles and GStreamer
Many possibilities: - xvimagesink + GstVideoOverlay - eglglessink - glimagesink - make our own sink
xvimagesink with overlay: not so fast, I suspect it triggers a download to CPU to get the image from xvimagesink's Drawable to a cairo-gl surface. eglglessink: promising, but obviously doesn't support openGL or GLX glimagesink started work on proper integration, likely will require addition of a few features there
First attempt:a cairosink that does everything
Cairosink: status
- GstAllocator and GstMemory subclasses - Use either OpenGL or OpenGLES - Possibility to use PixelBuffer Objects (PBOs) (inefficiently)
allocator and memory not that useful as it's the only element using it. Main advantage: _could_ allow to start uploading the frame as soon as possible. PBOs in a nutshell: a way to asynchronously upload a frame to a texture. The upload is then done through the magic of DMA, without the CPU being involved. More on why cairosink is currently inefficient at using it later. FIXME: graph explaining PBO?
Cairosink: simplistic benchmark
Synchronous / PBO / XV XV: limited by refresh rate? Looks like its genuine limit (often below 60) frames 720x400, RGB or YUV
Cairosink: difficulties (or rants)
- cairogles can be compiled with either gl or glesv2 (not both) - driver bugs and regressions - cairo only supports RGB - desktop vs mobile GPUs
driver regression: GL+EGL used to work on my machine...
Desktop vs mobile GPU
- EGLMakeCurrent() is costly on many mobile GPUs/drivers - bandwidth differences
context acquisition (makeCurrent) slowness is likely to counter the effects of asynchronous direct upload
Cairosink: what's to do - more flexibility on code path - compatibility with other elements (not our own memory/allocator)
we want to be able to tailor to various GPUs various "options": is it faster to do a synchronous or asynchronous upload?
Cairosink: other ideas to explore - libgstgl backend (ongoing experiment) - wrapper around glimagesink
libgstgl issues: mechanism to MakeCurrent() or save/restore states share GMainContext? (all in one thread)
Thank youQuestions?