Download doc/tutorials/introduction/android_binary_package/android_ocl_intro.markdown from tgollogly/test: direct link, hf CLI and curl.
- Browser
- Download file 13.8 kB
-
https://huggingface.co/tgollogly/test/resolve/main/doc/tutorials/introduction/android_binary_package/android_ocl_intro.markdown
- Command line
-
hf download hf://tgollogly/test/doc/tutorials/introduction/android_binary_package/android_ocl_intro.markdown
-
curl -L -o android_ocl_intro.markdown https://huggingface.co/tgollogly/test/resolve/main/doc/tutorials/introduction/android_binary_package/android_ocl_intro.markdown
13.8 kB
| Use OpenCL in Android camera preview based CV application {#tutorial_android_ocl_intro} | |
| ===================================== | |
| @prev_tutorial{tutorial_android_dnn_intro} | |
| @next_tutorial{tutorial_macos_install} | |
| | | | | |
| | -: | :- | | |
| | Original author | Andrey Pavlenko, Alexander Panov | | |
| | Compatibility | OpenCV >= 4.9 | | |
| This guide was designed to help you in use of [OpenCL ™](https://www.khronos.org/opencl/) in Android camera preview based CV application. | |
| Tutorial was written for [Android Studio](http://developer.android.com/tools/studio/index.html) 2022.2.1. It was tested with Ubuntu 22.04. | |
| This tutorial assumes you have the following installed and configured: | |
| - Android Studio (2022.2.1.+) | |
| - JDK 17 | |
| - Android SDK | |
| - Android NDK (25.2.9519653+) | |
| - download OpenCV source code from [github](git@github.com:opencv/opencv.git) or from [releases](https://opencv.org/releases/) and build by [instruction on wiki](https://github.com/opencv/opencv/wiki/Custom-OpenCV-Android-SDK-and-AAR-package-build). | |
| It also assumes that you are familiar with Android Java and JNI programming basics. | |
| If you need help with anything of the above, you may refer to our @ref tutorial_android_dev_intro guide. | |
| This tutorial also assumes you have an Android operated device with OpenCL enabled. | |
| The related source code is located within OpenCV samples at | |
| [opencv/samples/android/tutorial-4-opencl](https://github.com/opencv/opencv/tree/4.x/samples/android/tutorial-4-opencl/) directory. | |
| How to build custom OpenCV Android SDK with OpenCL | |
| -------------------------------------------------- | |
| 1. __Assemble and configure Android OpenCL SDK.__ | |
| The JNI part of the sample depends on standard Khornos OpenCL headers, and C++ wrapper for OpenCL and libOpenCL.so. | |
| The standard OpenCL headers may be copied from 3rdparty directory in OpenCV repository or you Linux distribution package. | |
| C++ wrapper is available in [official Khronos reposiotry on Github](https://github.com/KhronosGroup/OpenCL-CLHPP). | |
| Copy the header files to didicated directory in the following way: | |
| @code{.bash} | |
| cd your_path/ && mkdir ANDROID_OPENCL_SDK && mkdir ANDROID_OPENCL_SDK/include && cd ANDROID_OPENCL_SDK/include | |
| cp -r path_to_opencv/opencv/3rdparty/include/opencl/1.2/CL . && cd CL | |
| wget https://github.com/KhronosGroup/OpenCL-CLHPP/raw/main/include/CL/opencl.hpp | |
| wget https://github.com/KhronosGroup/OpenCL-CLHPP/raw/main/include/CL/cl2.hpp | |
| @endcode | |
| libOpenCL.so may be provided with BSP or just downloaded from any OpenCL-cabaple Android device with relevant arhitecture. | |
| @code{.bash} | |
| cd your_path/ANDROID_OPENCL_SDK && mkdir lib && cd lib | |
| adb pull /system/vendor/lib64/libOpenCL.so | |
| @endcode | |
| System verison of libOpenCL.so may have a lot of platform specific dependencies. `-Wl,--allow-shlib-undefined` flag allows | |
| to ignore 3rdparty symbols if they are not used during the build. | |
| The following CMake line allows to link the JNI part against standard OpenCL, but not include the loadLibrary into | |
| application package. System OpenCL API is used in run-time. | |
| @code | |
| target_link_libraries(${target} -lOpenCL) | |
| @endcode | |
| 2. __Build custom OpenCV Android SDK with OpenCL.__ | |
| OpenCL support (T-API) is disabled in OpenCV builds for Android OS by default. | |
| but it's possible to rebuild locally OpenCV for Android with OpenCL/T-API enabled: use `-DWITH_OPENCL=ON` option for CMake. | |
| You also need to specify the path to the Android OpenCL SDK: use `-DANDROID_OPENCL_SDK=path_to_your_Android_OpenCL_SDK` option for CMake. | |
| If you are building OpenCV using `build_sdk.py` please follow [instruction on wiki](https://github.com/opencv/opencv/wiki/Custom-OpenCV-Android-SDK-and-AAR-package-build). | |
| Set these CMake parameters in your `.config.py`, e.g. `ndk-18-api-level-21.config.py`: | |
| @code{.py} | |
| ABI("3", "arm64-v8a", None, 21, cmake_vars=dict('WITH_OPENCL': 'ON', 'ANDROID_OPENCL_SDK': 'path_to_your_Android_OpenCL_SDK')) | |
| @endcode | |
| If you are building OpenCV using cmake/ninja, use this bash script (set your NDK_VERSION and your paths instead of examples of paths): | |
| @code{.bash} | |
| cd path_to_opencv && mkdir build && cd build | |
| export NDK_VERSION=25.2.9519653 | |
| export ANDROID_SDK=/home/user/Android/Sdk/ | |
| export ANDROID_OPENCL_SDK=/path_to_ANDROID_OPENCL_SDK/ | |
| export ANDROID_HOME=$ANDROID_SDK | |
| export ANDROID_NDK_HOME=$ANDROID_SDK/ndk/$NDK_VERSION/ | |
| cmake -GNinja -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake -DANDROID_STL=c++_shared -DANDROID_NATIVE_API_LEVEL=24 | |
| -DANDROID_SDK=$ANDROID_SDK -DANDROID_NDK=$ANDROID_NDK_HOME -DBUILD_JAVA=ON -DANDROID_HOME=$ANDROID_SDK -DBUILD_ANDROID_EXAMPLES=ON | |
| -DINSTALL_ANDROID_EXAMPLES=ON -DANDROID_ABI=arm64-v8a -DWITH_OPENCL=ON -DANDROID_OPENCL_SDK=$ANDROID_OPENCL_SDK .. | |
| @endcode | |
| Preface | |
| ------- | |
| Using [GPGPU](https://en.wikipedia.org/wiki/General-purpose_computing_on_graphics_processing_units) | |
| via OpenCL for applications performance enhancements is quite a modern trend now. | |
| Some CV algo-s (e.g. image filtering) run much faster on a GPU than on a CPU. | |
| Recently it has become possible on Android OS. | |
| The most popular CV application scenario for an Android operated device is starting camera in preview mode, applying some CV algo to every frame | |
| and displaying the preview frames modified by that CV algo. | |
| Let's consider how we can use OpenCL in this scenario. In particular let's try two ways: direct calls to OpenCL API and recently introduced OpenCV T-API | |
| (aka [Transparent API](https://docs.google.com/presentation/d/1qoa29N_B-s297-fp0-b3rBirvpzJQp8dCtllLQ4DVCY/present)) - implicit OpenCL accelerations of some OpenCV algo-s. | |
| Application structure | |
| --------------------- | |
| Starting Android API level 11 (Android 3.0) [Camera API](http://developer.android.com/reference/android/hardware/Camera.html) | |
| allows use of OpenGL texture as a target for preview frames. | |
| Android API level 21 brings a new [Camera2 API](http://developer.android.com/reference/android/hardware/camera2/package-summary.html) | |
| that provides much more control over the camera settings and usage modes, | |
| it allows several targets for preview frames and OpenGL texture in particular. | |
| Having a preview frame in an OpenGL texture is a good deal for using OpenCL because there is an | |
| [OpenGL-OpenCL Interoperability API (cl_khr_gl_sharing)](https://www.khronos.org/registry/cl/sdk/1.2/docs/man/xhtml/cl_khr_gl_sharing.html), | |
| allowing sharing OpenGL texture data with OpenCL functions without copying (with some restrictions of course). | |
| Let's create a base for our application that just configures Android camera to send preview frames to OpenGL texture and displays these frames | |
| on display without any processing. | |
| A minimal `Activity` class for that purposes looks like following: | |
| @code{.java} | |
| public class Tutorial4Activity extends Activity { | |
| private MyGLSurfaceView mView; | |
| @Override | |
| public void onCreate(Bundle savedInstanceState) { | |
| super.onCreate(savedInstanceState); | |
| requestWindowFeature(Window.FEATURE_NO_TITLE); | |
| getWindow().setFlags(WindowManager.LayoutParams.FLAG_FULLSCREEN, | |
| WindowManager.LayoutParams.FLAG_FULLSCREEN); | |
| getWindow().setFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON, | |
| WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON); | |
| setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE); | |
| mView = new MyGLSurfaceView(this); | |
| setContentView(mView); | |
| } | |
| @Override | |
| protected void onPause() { | |
| mView.onPause(); | |
| super.onPause(); | |
| } | |
| @Override | |
| protected void onResume() { | |
| super.onResume(); | |
| mView.onResume(); | |
| } | |
| } | |
| @endcode | |
| And a minimal `View` class respectively: | |
| @snippet samples/android/tutorial-4-opencl/src/org/opencv/samples/tutorial4/MyGLSurfaceView.java minimal_surface_view | |
| @note we use two renderer classes: one for legacy [Camera](http://developer.android.com/reference/android/hardware/Camera.html) API | |
| and another for modern [Camera2](http://developer.android.com/reference/android/hardware/camera2/package-summary.html). | |
| A minimal `Renderer` class can be implemented in Java (OpenGL ES 2.0 [available](http://developer.android.com/reference/android/opengl/GLES20.html) in Java), | |
| but since we are going to modify the preview texture with OpenCL let's move OpenGL stuff to JNI. | |
| Here is a simple Java wrapper for our JNI stuff: | |
| @snippet samples/android/tutorial-4-opencl/src/org/opencv/samples/tutorial4/NativePart.java native_part | |
| Since `Camera` and `Camera2` APIs differ significantly in camera setup and control, let's create a base class for the two corresponding renderers: | |
| @code{.java} | |
| public abstract class MyGLRendererBase implements GLSurfaceView.Renderer, SurfaceTexture.OnFrameAvailableListener { | |
| protected final String LOGTAG = "MyGLRendererBase"; | |
| protected SurfaceTexture mSTex; | |
| protected MyGLSurfaceView mView; | |
| protected boolean mGLInit = false; | |
| protected boolean mTexUpdate = false; | |
| MyGLRendererBase(MyGLSurfaceView view) { | |
| mView = view; | |
| } | |
| protected abstract void openCamera(); | |
| protected abstract void closeCamera(); | |
| protected abstract void setCameraPreviewSize(int width, int height); | |
| public void onResume() { | |
| Log.i(LOGTAG, "onResume"); | |
| } | |
| public void onPause() { | |
| Log.i(LOGTAG, "onPause"); | |
| mGLInit = false; | |
| mTexUpdate = false; | |
| closeCamera(); | |
| if(mSTex != null) { | |
| mSTex.release(); | |
| mSTex = null; | |
| NativeGLRenderer.closeGL(); | |
| } | |
| } | |
| @Override | |
| public synchronized void onFrameAvailable(SurfaceTexture surfaceTexture) { | |
| //Log.i(LOGTAG, "onFrameAvailable"); | |
| mTexUpdate = true; | |
| mView.requestRender(); | |
| } | |
| @Override | |
| public void onDrawFrame(GL10 gl) { | |
| //Log.i(LOGTAG, "onDrawFrame"); | |
| if (!mGLInit) | |
| return; | |
| synchronized (this) { | |
| if (mTexUpdate) { | |
| mSTex.updateTexImage(); | |
| mTexUpdate = false; | |
| } | |
| } | |
| NativeGLRenderer.drawFrame(); | |
| } | |
| @Override | |
| public void onSurfaceChanged(GL10 gl, int surfaceWidth, int surfaceHeight) { | |
| Log.i(LOGTAG, "onSurfaceChanged("+surfaceWidth+"x"+surfaceHeight+")"); | |
| NativeGLRenderer.changeSize(surfaceWidth, surfaceHeight); | |
| setCameraPreviewSize(surfaceWidth, surfaceHeight); | |
| } | |
| @Override | |
| public void onSurfaceCreated(GL10 gl, EGLConfig config) { | |
| Log.i(LOGTAG, "onSurfaceCreated"); | |
| String strGLVersion = GLES20.glGetString(GLES20.GL_VERSION); | |
| if (strGLVersion != null) | |
| Log.i(LOGTAG, "OpenGL ES version: " + strGLVersion); | |
| int hTex = NativeGLRenderer.initGL(); | |
| mSTex = new SurfaceTexture(hTex); | |
| mSTex.setOnFrameAvailableListener(this); | |
| openCamera(); | |
| mGLInit = true; | |
| } | |
| } | |
| @endcode | |
| As you can see, inheritors for `Camera` and `Camera2` APIs should implement the following abstract methods: | |
| @code{.java} | |
| protected abstract void openCamera(); | |
| protected abstract void closeCamera(); | |
| protected abstract void setCameraPreviewSize(int width, int height); | |
| @endcode | |
| Let's leave the details of their implementation beyond of this tutorial, please refer the | |
| [source code](https://github.com/opencv/opencv/tree/4.x/samples/android/tutorial-4-opencl/) to see them. | |
| Preview Frames modification | |
| --------------------------- | |
| The details OpenGL ES 2.0 initialization are also quite straightforward and noisy to be quoted here, | |
| but the important point here is that the OpeGL texture to be the target for camera preview should be of type `GL_TEXTURE_EXTERNAL_OES` | |
| (not `GL_TEXTURE_2D`), internally it keeps picture data in _YUV_ format. | |
| That makes unable sharing it via CL-GL interop (`cl_khr_gl_sharing`) and accessing its pixel data via C/C++ code. | |
| To overcome this restriction we have to perform an OpenGL rendering from this texture to another regular `GL_TEXTURE_2D` one | |
| using _FrameBuffer Object_ (aka FBO). | |
| ### C/C++ code | |
| After that we can read (_copy_) pixel data from C/C++ via `glReadPixels()` and write them back to texture after modification via `glTexSubImage2D()`. | |
| ### Direct OpenCL calls | |
| Also that `GL_TEXTURE_2D` texture can be shared with OpenCL without copying, but we have to create OpenCL context with special way for that: | |
| @snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp init_opencl | |
| Then the texture can be wrapped by a `cl::ImageGL` object and processed via OpenCL calls: | |
| @snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp process_pure_opencl | |
| ### OpenCV T-API | |
| But instead of writing OpenCL code by yourselves you may want to use __OpenCV T-API__ that calls OpenCL implicitly. | |
| All that you need is to pass the created OpenCL context to OpenCV (via `cv::ocl::attachContext()`) and somehow wrap OpenGL texture with `cv::UMat`. | |
| Unfortunately `UMat` keeps OpenCL _buffer_ internally, that can't be wrapped over either OpenGL _texture_ or OpenCL _image_ - so we have to copy image data here: | |
| @snippet samples/android/tutorial-4-opencl/jni/CLprocessor.cpp process_tapi | |
| @note We have to make one more image data copy when placing back the modified image to the original OpenGL texture via OpenCL image wrapper. | |
| Performance notes | |
| ----------------- | |
| To compare the performance we measured FPS of the same preview frames modification (_Laplacian_) done by C/C++ code (call to `cv::Laplacian` with `cv::Mat`), | |
| by direct OpenCL calls (using OpenCL _images_ for input and output), and by OpenCV _T-API_ (call to `cv::Laplacian` with `cv::UMat`) on _Sony Xperia Z3_ with 720p camera resolution: | |
| * __C/C++ version__ shows __3-4 fps__ | |
| * __direct OpenCL calls__ shows __25-27 fps__ | |
| * __OpenCV T-API__ shows __11-13 fps__ (due to extra copying from `cl_image` to `cl_buffer` and back) | |