Private mobile device farm as an indicator of mobile testing culture
Have you ever heard of Spiral Dynamics? It’s a sociological theory about the cultural levels of a society. There is a model of the evolutionary development of individuals, organizations, and societies based on it. And the concepts behind this model can also be applied to any organization or process.
This article will demonstrate how to level up the culture of mobile testing by introducing in-house distributed mobile farms into the mobile development process. It will explore the types of situations that suit this kind of farm compared to cloud mobile device farms. It will also cover the important aspects of automation that are critical when setting up and configuring this type of solution.
Spiral dynamics model for mobile testing
Let’s apply it for the mobile development process and in particular to mobile testing:
-
Beige/Survival
There is no dedicated testing team. Mobile developers do unsystematic testing on their own mobile phones or emulators/simulators(*). May suit only small or simple applications.(*) The terms emulator and simulator will be used interchangeably in the article because the difference is not essential for this topic.
-
Purple/Security
There is no dedicated testing team. Developers have a suite of automated tests at the unit/component level on their own machines and devices and do unsystematic acceptance testing before release. -
Red/Power
A test engineer appears in the team. Some target devices are now planned and dedicated for testing purposes. Acceptance testing is done by test plan or checklists. Unit tests are executed irregularly on test devices by the test engineer. -
Blue/Order
A testing team is gathered. CI is established and maintained by this team or a dedicated DevOps engineer. Unit tests run regularly on CI with emulators. Real devices are bought according to plan and there is a specific procedure of device allocation. Testing on real devices is manual but usually scripted in test scenarios. First regression automated tests are created. -
Orange/Success
Test automation is added to the process. Some dedicated devices are added to CI. Mostly, automated testing is still on emulators and only smoke tests are done on dedicated devices. Occasionally full regression is done on real devices. Team members interchange devices when they need to check some specifics. Cloud mobile farms are used on demand. Special third-party services are used to gather information from user devices (e.g. to register app crashes or to calculate usage statistics). -
Green/Community
Cloud mobile farms are primarily used for automated testing. CI delegates regression test runs on cloud devices or uses emulators to run smoke tests on every commit. Small numbers of in-house devices are used for manual acceptance testing and defect troubleshooting. There is the added possibility of doing A/B testing on user devices but switching new features on for only a small target group. -
Yellow/Synergy
In-house device farms are used for most types of testing. Both simulators and real devices are included in such local farms. Cloud farms are used for compatibility testing and discovering specific cases on rare devices. Team members rarely need a device on the table and allocate it in a local lab with remote access to it. Even if a device is placed on an engineer’s desk it can still stay a part of the hub and belong to the farm. -
Turquoise/Holism
The future of mobile testing?
The in-house distributed device farm
At Grid Dynamics we have been involved in mobile testing for many years. Primarily in test automation scenarios for complex mobile projects that have dozens of screens, a number of integrations, and hundreds of features. We usually operate at levels from Blue to Green. Sometimes we also operate at the Red level, for example at the beginning of a complex project or when the project is tiny and there is no need for a complex testing approach. Recently we’ve discovered how to reach the next (Yellow) level by investigating how to build a local mobile farm based on an open-source platform. We did not however just stop at the investigative stage, and went on to create a complete automated solution in order to set up, configure, and scale the local farm.
Comparison of private and cloud solution
| Private | Cloud | |
| Initial setup costs How much money you need to set up the solution. | MEDIUM You need to: (a) buy devices (most expensive part); (b) buy additional hosts for the lab; (c) invest in initial setup work. | LOW Usually you do not need to pay anything unless you provide your own devices to the cloud farm. In this case the cost is comparable with private farms. |
| Usage costs How much money you spend while operating the farm services. | LOW Primarily only electricity costs. The more tests you run the lower price you will pay, and the more efficient your overall investment will be. | HIGH Depends on plan used, number of devices, number of tests, and number of hours. The more tests you run, the more you pay. |
| Maintenance costs How much money you need to support the solution. | LOW/MEDIUM Small localized lab maintenance costs are comparable with the costs of unsystematic device usage by your team. If you have a big distributed farm you probably need a dedicated lab engineer to manage the overall stability of the system including device clean up etc. | LOW Maintenance costs are usually covered as part of the overall tariff plan. However, if you have delegated your devices you need to be ready to potentially also pay costs associated with upgrading and repairing devices. |
| Location restrictions Ability to keep your application and data within the bounds of allowable regions. | HIGH The nodes in the private farm are usually located in your target region or country. This means that data and applications are rarely located outside any allowable regions. As it’s possible to set up the in-house farm on the same cluster where the backend is located, there shouldn’t be any issues with internal access or integrations with third party services. | LOW The lab is usually located in an external region (e.g. Eastern Europe, Asia or South America). In some cases this may mean that some third party services are not accessible from that location. There may also be certain restrictions that prevent applications or data being accessible from those countries or regions. |
| Security Access rights to your application, data, and backend environments. | HIGH It’s possible to integrate the lab access with corporate LDAP or SSO and use groups to restrict access to applications, services, and reports based on it. As well as regional segregation as described above, the lab could be within the same local network or demilitarized zone with the backend so no data leaks are likely. | MEDIUM To divide the access to different applications and data you probably need to have different accounts and pay for each of them. Access to the backend can sometimes be tricky through ssh-tunnels and require organizational and administrational approvals. This makes it very difficult to integrate with internal LDAP and groups. |
| Development speed How much time is needed to investigate causes of a failure, troubleshoot a failing test, or debug a test/feature during development. | HIGH Debugging of tests or issues can be easily switched between a real device or emulator and done before merging code into a repository. No limitation of test runs. According to our experience, sometimes it is up to 10x faster than using cloud farms. | MEDIUM/LOW There are a number of factors that contribute to making the feedback loop slow or inconvenient. This is due to the primary development being done on simulators and only later then tried on real devices using the full regression suite with delayed analysis of failures. |
| Device diversification How many different device models and configurations are provided. | LOW The best practice here is to concentrate on a limited number of the most usable devices to speed up development of applications and testing. It is better to use cloud farms for compatibility testing. The more diversity you want in your local lab the higher price you will have to pay. | HIGH Usually, one of the main advantages of cloud farms is the ability to provide different devices and configurations to cover specific cases or to do compatibility testing. If such testing is properly planned there should not be any issues and usage of clouds farms is recommended. |
| Device availability How much time is needed to allocate to the required devices for testing. | HIGH Proper planning of local lab resources as well as the analytical capabilities of device usage and idle time gives you everything you need to always have devices ready to use even for emergency investigation of production issues. Even if you need to have the device on your table to interact with it, the device can still be a part of the lab and used when it’s free. | MEDIUM Sometimes it can be tricky to have access to a specific device as quickly as desired or it can be affected by previous states that require additional steps to clean-up the device. The inability to interact with devices as needed can be an issue meaning you then may have to have spare devices on your table that are not part of the lab. |
| Stability of results The ability to have predictable and repeatable test results. | HIGH The test results are highly predictable because the same devices can be used for the same kinds of tests. You can fully manage who, when, and for how long you allocate devices for exclusive usage. | MEDIUM Can be high when you use only a limited set of devices. But it makes mincemeat of the main advantage of the cloud farm – device variety. Oftentimes when testing, “any” device is selected, which can lead to unpredictable failures of autotests. This can be caused by non-functional reasons (slow device, network, or lower screen resolution). |
Grid Dynamics accelerator for private mobile farms
Within Grid Dynamics, we’ve elaborated on the Mobile Device Farms Accelerator (MDFA). We consider it as the ability to have your personal managed cloud of devices. The main benefits it provides are speeding up development and testing, easier CI integration, and lower costs in comparison with cloud device farms.
We based our MDFA on the open-source solution from qaprosoft, that in turn is based on the open-source platform OpenSTF.
Being adept at automation, we added the following features to the solution:
- Ability to automatically setup and configure the main lab host node.
- Ability to automatically prepare and configure iOS and Android hosts.
- Ability to easily attach new devices to the farm.
- Ability to verify the farm configuration and working status by running special automated verification checks.
- Ability to easily integrate with CI (Jenkins).
- Ability to create automated mobile tests that can switch between emulators and real devices in your private farm and cloud farm. It is as simple as changing configuration parameters.
- Ability to authenticate with LDAP and use LDAP groups to manage access to resources.
- Ability to automatically gather device usage statistics.
Look at the picture below. This is an example of how the distributed lab could be organized. In this configuration, for instance, if the application or data has restrictions to be used only in the EU, testing with real data could be done only in Krakow’s lab subdivision. However, the development can still go ahead in Kharkiv using synthetic data. And if there is an incident, developers from the Kharkiv team can reach the device and data located in Krakow to help resolve the issue.
Moreover, in the current reality, COVID-19 has caused changes to the previous regime of work. Where previously most work regimes involved co-located team members, this has rapidly transitioned to remote work from home regimes. This makes mobile development more challenging because you are not able to just give the device to your teammate who is sitting at the neighboring desk. But our approach allows every team member to join the lab from their work location and share devices with others.
Cloud farms are still helpful
It’s important to note there are still scenarios where you may need to use cloud farms for different kinds of tests. But you can do it more effectively by paying the same or lower costs. How is that possible?
For example, you may have a limited budget to test on cloud farms and you need to manage the following activities:
- Running compatibility tests.
- Troubleshooting issues on specific configurations.
- Performing acceptance testing on targeted devices.
- Debugging new features and tests.
- Running automated smoke tests.
- Running an automated regression test suite.
- Doing other types of tests (e.g. performance or usability).
So, to remain within the budget you have to either run more tests on emulators (e.g. #4, #5 and #6 from the list above); use devices at your table (#3 and #7), or decrease the number of configurations for compatibility testing. All these approaches work but they are mostly compromises.
The introduction of an in-house mobile farm can help you to properly manage your devices (you already have them for development needs) and unify the testing approach whether it’s a simulator or real device. This covers a real device in a local farm or cloud farm, manual or automated testing, and those initiated by humans or from CI. In all those cases you just need to choose the configuration, “book” the device, and run the tests. Through proper planning it’s possible to move most parts of the activities to the local device farm and keep running tests on real devices when needed. That will either decrease the budget you need to spend on cloud device providers or increase the number of experiments and configurations you can do there.
Conclusion
The approach described in the article will help to increase the quality level of your mobile applications by increasing the culture level of testing. Mostly you’ll either do the same test activities or even include new types into your portfolio. In addition you’ll involve all members in testing and distribute the tests between the appropriate platforms. It’ll help to increase the number of test runs as well as save money.
So what is the last ‘Turquoise’ level? Let’s dream it up:
-
Turquoise/Holism
Cloud farms and in-house farms share the same platform and can be joined into the same infrastructure. Users provide their device as a part of a testing lab. The development process assumes a continuous delivery paradigm and new features are tested on the target group in production. The testing team coordinates all the test types. Everyone can run tests on the provided infrastructure at any time. This also includes uses where team members can run tests on their own device if something is wrong and report the results back to the development team.
Is it possible to get there now or even tomorrow? Not quite. But the day after tomorrow – why not? But it does require a joint effort among mobile developers, cloud farm providers, business, and customers. Usage of private farms today is a required prerequisite of such a future and we know how to do this step right now. Feel free to contact us regarding the mobile development, testing, and set up of your private mobile farm.
