Let ijkplayer support the method of inserting custom GPU filter

Recently, due to work reasons, we need to provide an example of inserting our aiyaeffects SDK into ijkplayer, so we have to look at the ijkplayer code. There is no interface for setting custom GPU filters in ijkplayer, so you can only do it yourself in the end. I have to say that the open source ijkplayer player of BiliBili is indeed very powerful. The code design is very clear. If you look carefully, you can learn a lot.

Ijkplayer source code acquisition and compilation method

The source code address can be compiled by referring to readme:

Ijkplayer analysis and modification

In the ijkplayer example project for Android, the video playing interface is tv.danmaku.ijk.media.example.activities.videoactivity. In the videoactivity, ijkvideoview is used to play video, which is located under the tv.danmaku.ijk.media.example.widget.media package.

The use of ijkvideoview is basically the same as that of Android videoview. In ijkvideoview, set the video source and call the setvideouri method, which will call the openvideo method of private property. In the openvideo method, an imediaplayer will be created according to the value of msettings. Getplayer():

As can be seen from the above code, msettings are used in many places in ijkvideoview. Msettings are mainly set by users and saved through SharedPreferences, including audio and video decoding settings, whether to use opensles, rendering view, etc. Please refer to the settingsactivity interface.

Create ijkmediaplayer according to playertype. The first two types are Google's exoplayer and Android's mediaplayer. In addition, it is the real ijkplayer created.

Other parameters in msettings will eventually be converted and set through the setoption method of ijkmediaplayer, and ijkmediaplayer.setoption directly calls the native method. Entering ijkmediaplayer, you can find that many methods in ijkmediaplayer are native methods or call native methods.

Add setglfilter interface

Search one of the methods globally under the ijkmedia folder_ Setdatasource, the contents are as follows:

That is, in the Android version of ijkplayer, the entry is ijkmedia \ ijkplayer \ Android \ ijkplayer_ jni.c

In ijkmediaplayer.java and ijkplayer_ Add setglfilter method in JNI. C file:

Callback the glfilter related methods set by the user during rendering

In the ijkmedia / ijksdl / gles2 / internal. H file, ijk_ GLES2_ In the renderer structure:

These three methods are the three methods of ijkflter. They will correspond to the three methods of ijkfilter object set in Java in JNI.

Global search gldraw found only one ijk in renderer. C_ GLES2_ Renderer_ In the renderhoverlay method, the rendering work is also performed by this method. Of course, when ijkplayer uses OpenGLES to render, it will render according to the specific format of the data decoded from the video, such as yuv420p, yuv420sp, rbg565 and many other formats. The details can be found under ijkmedia / ijksdl / gles2 /_ RGB. C \ renderer. Yuv420p. C and so on.

When the user sets glfilter in the Java layer, the three methods of glfilter should be called back by C when appropriate. As can be seen from the name, these three methods are actually the same as the three methods defined in the glsurfaceview. Renderer interface.

Specific in ijk_ GLES2_ Renderer_ The modifications in the renderhoverlay method are as follows:

Glfilter implementation analysis from setting to calling

The above interface has been written, and the implementation has also been written. Now, you need to correspond the three methods of glfilter passed in by the interface with the three methods executed, so that the user's filter can really play a role.

IJK_ GLES2_ The renderer is created according to SDL_ From vouthoverlay, looking for SDL_ Where does Vout overlay come from? You can search ijkmedia / ijksdl / Android / ijksdl all the way_ vout_ android_ Func in nativewindow. C_ create_ Overlay method:

SDL_ Creation and SDL of voutaoverlay_ About Vout, find SDL again_ The source of Vout can be found

ijkmedia/ijksdl/android/ijksdl_ vout_ android_ SDL in nativewindow. C_ VoutAndroid_ CreateForANativeWindow：

There is nothing to associate with its initialization, so we can only find the place to call it. SDL found after search_ VoutAndroid_ Createforanativewindow is only available in ijkmedia \ ijksdl \ Android \ ijksdl_ vout_ android_ SDL of surface. C_ VoutAndroid_ Createforandroidsurface method:

It seems to be covered with a layer of skin. Then go on to find the SDL_ VoutAndroid_ Createforandroidsurface, search for ijkmedia \ ijkplayer \ Android \ ijkplayer calling it_ Ijkmp for Android. C_ android_ Create method:

You can see ijkmp_ android_ Create returns an ijkmediaplayer, which also has such a class in Java, so the dawn should not be far away.

Search ijkmp again_ android_ Create, and only ijkmedia \ ijkplayer \ Android \ ijkplayer calls it_ Ijkmediaplayer in JNI. C_ native_ The setup method. At this point, you can pass the ijkfilter.

In the above process, you can see from renderer. C to JNI and ijk_ GLES2_ The creation of renderer depends on SDL_ VoutOverlay。 SDL_ The creation of voutdoverlay depends on SDL_ Vout。 SDL_ Vout is a member of ffplayer, and ffplayer is a member variable of ijkmediaplayer. The ijkmediaplayer of the Java layer depends on ijkmediaplayer

From glfilter to ijk_ GLES2_ Renderer

According to the above analysis, in the setglfilter method added in JNI, we can pass the glfilter method to ijkmediaplayer - > ffplayer - > sdlvout, and then to SDL_ Voutdoverlay, and then by SDL_ Voutdoverlay passed to ijk_ GLES2_ Just use renderer. In this way, the function of adding filter is added, and the process of ijkplayer will not be affected, so that IOS can also quickly realize the function of adding GPU filter.

First in SDL_ Voutdoverlay and SDL_ The structure definition in Vout (ijkmedia / ijksdl / ijksdl_vout. H file) is also added in ijk_ GLES2_ Members added in renderer:

Then, in the above methods, complete these member data from SDL respectively_ Vout to SDL_ Voutdoverlay, then ijk_ GLES2_ The delivery of the renderer.

They are:

Ijk in ijkmedia \ ijksdl \ gles2 \ renderer. C file_ GLES2_ Renderer_ Create method

ijksdl\android\ijksdl_ vout_ android_ Func in nativewindow. C file_ create_ Overlay method

Finally, SDL needs to be completed_ Assign values to these members in Vout and call the relevant methods of glfilter object passed in from Java (in ijkplayer_jni. C file):

At this point, Java is connected to the renderer in SDL. The processing in Java is basically the same as setting the renderer in glsurfaceview. The difference is that the glfilter added to ijkplayer has provided an original video image as the parameter of ondrawframe. In glfilter, you only need to process the texture and render it.

Insert filter example

Recompile the modified code, and the compiled library will be automatically updated to the Android project of ijkplayer. After setting the custom filter, you can see the effect without accident. The following are the original video, the video after black-and-white filter processing, and the video effect with aiyaeffectssdk and special effects. Because CSDN limits the size of the picture, a short segment is intercepted:

The project is too big and there are not many modifications, so we don't upload the code. Friends in need can download the source code of ijkplayer according to the above process and modify it by themselves. At the same time, you can also see the source code of ijkplayer.

The above article on how to make ijkplayer support inserting custom GPU filters is all that Xiaobian has shared with you. I hope it can give you a reference and support more programming tips.