AI-driven game automation has been in practice in large-scale gaming companies lately. Games are complex, with a wide area to cover when testing. And with different playing tactics varying with each player, it is the most challenging part for a game tester to cover everything in the game environment.
A single QA involved in a game testing cannot mock 100 different players' way of playing. This is why most gaming companies do AI-driven automation to simulate every possible way of playing a certain game. However, automated testing is limited as it depends on AI's programming to be carried out effectively.
Game automation requires very specific planning and procedure, which can be different in different games. It may also take longer than expected to meet the requirements of automation. With very limited resources available online, it will be extensive self-research work for the QA department.
Areas to Assess before Automation Testing
Automation is such a thing in the software industry which always strikes with ROI, i.e. return on investment. So, is it always good to automate every game? And, is it always profitable to automate? Well, the answer is mostly no.
Let's look into the areas we should analyse before game automation testing.
Game evaluation stage
Since games are complex, a game analysis should be done ideally before or during the development phase. This helps to figure out what scenarios should & shouldn't be automated.
Time constraint
The time of the project is another vital area to consider before planning on the execution of game automation. If the project is bound to be completed in a few months, spending time and money on automation will likely result in waste as it wouldn't be used to its full potential.
Testing essentials provided by the company
Another question to consider is, "Can the company invest time and money in setting up the automation environment, team, tools and devices?" This is important because it is impossible to carry out automation testing if a company is unwilling to provide the SQA engineers with the essential tools required.
Determining the newness of the project
Is it a new project or an existing project? Placing this question up and forward will help determine the project's novelty and whether the automation testing is to be executed. When automation testing is carried out in new software, the chances of receiving high-quality results are more than that of applying it in an already existing, tested software.
Benefits and Drawbacks of Automation Testing
Here are some potential benefits and drawbacks to consider when deciding whether to automate the testing of your mobile game.
Benefits:
- It is faster and more efficient than manual testing, especially for large or complex games.
- It can be run repeatedly, making it easier to catch regressions (i.e., issues that reappear after being fixed).
- It can be run on multiple devices and configurations, helping to ensure that the game works properly on a wide range of hardware and software.
Drawbacks:
- It requires significant upfront setup and maintenance, including the creation and maintenance of test scripts.
- It may not be able to cover every possible scenario or user interaction, so manual testing may still be necessary.
- It can be brittle, breaking or producing false positives if the game's user interface or functionality changes.
Challenges in Mobile Games Automation
Mobile game automation is impossible with only general mobile app automation tools and tech. This is why it is more challenging than normal utility application automation. Here are some challenges you might face during mobile game automation.
Selection of proper strategy and tools
Test automation can be costly, so it is determined after the project's financial feasibility and requires detailed planning beforehand. Automation can handle business-critical components, control user processes, user scenarios and complex app scenarios, and build up a regression test suite. So, choosing the right approach and tool in the case of mobile games automation testing is a pain in the back as it requires way too many ticks in the checklist.
Accessing mobile game elements
The most popular method for accessing mobile game elements is utilising image recognition to pinpoint the element's location on the screen. But, the main difficulty of mobile game automation testing is that the games run on the OpenGL and ActiveX graphics containers. These containers provide direct screen access, bypassing the operating system level services, making it difficult to access elements of the game inside the container through regular test automation tools. A specific device's rotation and movement mechanism are tough to automate without extra resources.
Image references
One issue with using test automation for mobile games that involves image recognition is the need to locate and keep track of suitable reference images. This can be difficult because the characteristics of the reference images needed for different tests on different devices may vary, such as the size and cropping of the images.
The image references should be updated whenever the mobile game's UI is altered. It takes a lot of time and effort to create and maintain a database of image references.
Interpreting test results
As mobile games incorporate physics and other random features that conventional mobile applications do not, it is harder to evaluate the findings of automated tests for games than for regular mobile apps.
Unpredictability
Games are inherently unpredictable and may contain infinite gameplay scenarios and variations. So, SQA engineers may face their greatest challenge when automating game test scripts.
Test accuracy on virtual and real devices
Testing only on actual mobile devices makes any software reliable. Emulators and simulators cannot be used to anticipate the degree of accuracy in game code. Having faced these issues multiple times, I can say that automation on emulators or simulators cannot define the accuracy of the gameplay and its other functionalities.
Network stability
Testing the performance of a mobile application should include how the app functions with various network scenarios, including working offline, different Wi-Fi speeds, connection dropouts, and poor signal strength. It is important to evaluate the effect of changes in network bandwidth on the app's functionality, as this can significantly impact its performance. Including this in automation scripts can be challenging.
Game code reusability
Unlike other software applications, games are often built with new codes and different game engines, making it difficult to reuse them. This requires careful planning and a specific testing approach to create and maintain functions that hide the game object code details from automated tests to maximise test reusability.
Risking test quality
Mobile tests often produce false negatives due to environmental, device stability, and test-specific issues. Debugging these tests can be time-consuming and increase the risk of incomplete test coverage. Additionally, the availability of labs for testing on mobile devices and emulators can impact automation stability and cause project delays. This can be challenging for short-span projects. If we move into a bigger project that will run for a couple of years or more, the automation environment will be the best to rely on the quality output of the product.
My Suggestions
Features and function uncertainties of any web and mobile app make it difficult for the SQA team to start with automation testing. Without proper Software Requirements Specification (SRS), it cannot be defined exactly where we can or should apply automation testing. Understanding the flow of any app or software is very vital for an SQA team.
Additionally, the software code should be stable. If you start automation with an unsure codebase which keeps changing, it will only create confusion and a messy environment. Soon as you get information that the code is stable, start writing automation scripts for the required parts of the software.
Pay close attention to the risk analysis and challenges part of this blog and cautiously decide on automating mobile games from your perspective. If you finalise on not automating due to limited resources or time, but the management says otherwise, negotiate on the needed areas. Agreeing to your request will benefit all parties involved; if not, you can always look for other options.
Closing out
It is difficult but not impossible to automate software and data complications, including audio, cinematic effects in games, huge data and localisation. Usually, the time available for automation is very limited due to short release cycles.
Automation does have positive effects on software that lasts throughout the year. But, applying it for mobile testing also has some drawbacks that should be considered before beginning the process. Every software development starts after a feasibility study of its market. Likewise, an automation testing feasibility study is also a must before starting it. Save time and monetary investment to boycott redundancy.
Thank you for reading. Subscribe for more!