I have a OpenGL-based rendering pipeline for filtering images, which I now want to use to process videos as well.
On the one end if the pipeline is an
AVPlayer for fetching frames from a video file, on the other end is my preview view, backed by a
CAEAGLLayer. The rendering itself happens async on a different thread because it's quite expensive. The view is hooked to a
CADisplayLink that triggers a new async rendering on every screen refresh. When the pipeline is done rendering into the layer's renderbuffer, I'm calling
presentRenderbuffer: to show it on screen (in the rendering thread). Draw requests happening while a rendering is still in progress are ignored.
This works—however, I seem to be getting synchronization issues with the display refresh. When I set the
frameInterval of the display link to 1 (call every frame), I'm getting ~2 FPS in the end (actual view refreshes). If I'm setting it to 2 (call every other frame), I'm suddenly getting 15 FPS. Setting it to 4 drops the FPS down to 2 again.
My guess is that the async call to
presentRenderbuffer: happens "at the wrong moment" in the run loop and is either ignored by the system or delayed.
Now I want to know what's the best practice for displaying the results of async renderings in a view. All the examples and docs I could find only describe the single-threaded case.
In these cases it is best to use double buffering which in your case are 2 textures. The rendering of the video should be done on the FBO (frame buffer object) with an attached texture. Since the drawing is on a separate thread I suggest you to create the 2 textures on the main context, main thread then create a shared context on the other thread which can now access the 2 threads.
Now there is no sense to block the background thread since it is expected to be slow so what it will do is keep rendering to the texture then once done send the texture to the main thread (where you present the buffer) and continue drawing to the other texture. Then repeat the procedure.
The main thread should then check if it got a request to display a new texture and when it does it should draw it to the main buffer and present it. If you need to draw it at 60FPS (or any other constant) you can still do that but you will be redrawing the same texture.
Now just to be on the same side you should still probably do some locking mechanism. Since the background thread does the buffer swapping (sends the new texture and starts drawing to the previous one) it makes sense that there is a boolean value
swapLocked where the main thread will set it to
true just before it starts drawing and set it to
false once it is done with the texture. Now if the background thread is done drawing and the
true it should not continue drawing. In this situation continue the swap and the drawing once the
swapLocked is set to
false. You can override the setter to do that but be careful to continue the process on the background thread since the setter will most likely be called on the main thread.