Appium is a framework that is able to scale across different use cases with any problem. Various mobile games today are tested on Appium and the Image Recognition Platform. All in all, this is a very efficient and effortless way to create test cases for mobile games and graphic-based content. In this article, we will look at Appium features, how to set them up, and how to use them.
Why To Use Image Recognition with Appium
Image recognition technology is widely used by mobile game developers nowadays, as creating automated tests for their mobile games is very easy and handy with this technique. Not only for games, but the technique can also be used for native apps on Android and/or iOS. Hence, developers are not limited when they use Appium.
Creating tests and working with those tests on both platforms (Android and iOS) is the USP of Appium Image Recognition. Test automation frameworks generally cannot recognize UI components such as OpenGL ES contents, their IDs, attributes, etc. Even if a human is clearly able to see the elements, test automation frameworks fail to do so.
This is why Appium Image Recognition is preferred. It can easily recognize all the graphical interface elements and test them.
Required Components of Appium Image Recognition
The following components and/or software are required in order to get started with Appium Image Recognition:
Which is a web service to run tests on real mobile devices in the cloud.
OpenCV:
OpenCV is a machine learning software as well as an open-source computer vision, and it provides a common infrastructure for computer vision applications to accelerate the use of machine perception.
Bitbar Akaze:
This is a fork of Accelerated KAZE with added support for JSON.
Jsoncpp:
Jsoncpp provides you with C++ support. It is used to represent numbers, ordered sequences of values, strings, etc. This is integrated with the Bitbar Akaze library.
Maven:
This is required to compile the whole code and launch it for use.
These components, along with Bitbar Public Cloud, are what you require to run tests and execute them in a well-timed manner.
Example: Using Java for Appium
Before starting, just take a look at Appium on GitHub. This will provide an example of Java using OpenCV and Akaze to use Appium.
This example project can be used by you to perform local tests or to understand the Bitbar cloud. You will also learn the difference between client-side and server-side execution.
Installing and Setting Up OpenCV
The OpenCV java libraries can be found inside the project in the ./lib//opencv directory. If you want to install all the libraries within Maven locally, type in the following command:
mvn install:install-file -Dfile=lib//opencv/opencv-2413.jar -DgroupId=opencv -DartifactId=opencv -Dversion=2.4.13 -Dpackaging=jar
Take note that every platform has a different version of libraries, so the above command is modified accordingly.
Installing and Setting Up Bitbar Akaze Fork
Bitbar Akaze Fork can be used optionally to implement AKAZE C++. The desired./lib//akaze/ folder with pre-built binaries for various operating systems is already present in the sample project on GitHub. Other things are fetched by Maven automatically.
Additional Information on OpenCV and Bitbar AKAZE Fork
The above example is dependent on the AKAZE algorithm to find out the matching key points between two images and then store them in a JSON file. The example also uses OpenCV Java to process the JSON files with key points to find homography.
AkazeImageFinder.java class will run akaze_match, and then it will use the key points from JSON so that it can identify a given image inside a scenery and return the results found.
In this example, we will be using a specific image from the ./queryimages/folder.
After the image is found, the test script can be tapped on, dragged, or interacted with in the game.
This class also has various functions to work with and manipulate the image captured. Some examples of this function are:
// returns the corners of the image if found, or null if no image was found
public Point[] findImage(String image, String scene)
// takes a screenshot, then tries to find the image on it and return the corners
public Point[] findImageOnScreen(String image)
// takes a screenshot, tries to find the image on the screen, then tap the middle of it
public void tapImageOnScreen(String image)
// takes a screenshot, finds the image on the screen, taps and holds it, then swipes the screen while still holding it
public void swipeScreenWithImage(String image, int repeats)
// takes a screenshot, tries to find the image, taps it and drags it to a certain position given as x and y offsets from middle of the screen
public void dragImage(String image, double x_offset, double y_offset)
Running Tests as Server-Side Run on Bitbar Public Cloud
First of all, you will need to create a project so that Appium can run on the Bitbar Public Cloud. You can do this by manually logging in and creating the project, or you can utilize the Bitbar API for creating your project. Using this code with curl will do the task easily:
$ curl
-X POST
-d “name=NewProject”
-H “Authorization: Bearer abcdefgh-1234-ijkl-m5n6-opqrstuvxyxz”
One more thing which you need to authorize before running your test is the project type and how you want to execute your test run. You can use the Appium server-side for Android or iOS, depending on what device you want to run the test on.
After the creation of the project, you can use the following script to create a test zip from your project:
./createAndroidZip.sh
// or alternatively …
./createiOSZip.sh
Since you have your test zip and application file available now, you can create test runs for your projects.
Just upload the app and the test zip to the Bitbar public cloud, determine the device group, and adjust the timeout of your test.
You need to use this code if you want to change the name of your Android or iOS test class:
# Name of the desired test suite and optionally specific test case, eg: AndroidSample#mainPageTest
TEST=${TEST:=”AndroidSample#mainPageTest”}
# JUnit file won't have the #caseName ending
JUNIT=”AndroidSample”
Optional: Building Up The Latest Libraries For Appium
To do this, first, we need to compile the latest version of jsoncpp.
Jsoncpp
To install jsoncpp, the following command is used:
git clone https://github.com/open-source-parsers/jsoncpp.git
cd jsoncpp
python amalgamate.py
This step will generate the following files, and new AKAZE binaries are needed to be compiled.
dist/jsoncpp.cpp: source file that is added to your project.
dist/json/json.h: corresponding header file for use in your project.
dist/json/json-forwards.h: header that provides forward declaration of all JsonCpp types.
After this, you also need to build and copy libjsoncpp.a library to the local directory:
mkdir build
cd build
cmake ..
make
cp src/lib_json/libjsoncpp.a /usr/local/lib/
OpenCV
To build a new library in OpenCV, use the following command:
git clone https://github.com/Itseez/opencv.git
cd opencv
git checkout 2.4
mkdir build
cd build/
cmake -G “Unix Makefiles” ..
make -j8
make install
This will create Java jar files, which will be used for image recognition samples. Jar files are installed by the command:
mvn install:install-file
-Dfile=path/to/opencv/build/bin/opencv-2413.jar
-DgroupId=opencv
-DartifactId=opencv
-Dversion=2.4.13
-Dpackaging=jar
The pom.xml file will need to be updated with every different OpenCV version.
Bitbar AKAZE
Bitbar Akaze can be built using the following commands:
git clone https://github.com/bitbar/akaze.git
cd akaze
# copy previously built versions of jsoncpp to ‘src/lib/jsoncpp/’, ‘src/lib/json/’ and src/lib/jsoncpp.cpp
mkdir ./src/lib/jsoncpp/
cp -r path/to/jsoncpp/dist/json ./src/lib/jsoncpp/
cp -r path/to/jsoncpp/dist/jsoncpp.cpp ./src/lib/
mkdir build
cd build
cmake ..
make
cp bin/akaze_* path/to/image-recognition/akaze/bin/
This is it. Using this guide, you will be able to run Appium Image Recognition and run your tests smoothly in your preferred environment.