Request a Demo

We’ll pull you off the monotonous, maddening assembly line of manual testing.

We’ve got your request. We’ll email shortly with more info.
Please provide the required information.
* Required fields must be completed
Mesmer logo

The bots are coming to take your job writing UI tests (thank goodness!)

April 2, 2020

I remember when I first learned about the idea that a machine — a robot — could take your job.

I was 10 years old, at a bowling alley, marveling at how quickly and precisely the machine at the other end of the alley would return the pins to their exact, proper location after each frame. It came up in conversation that there used to be people who did this job manually, then return the ball to the bowler.

These robotic machines are ubiquitous today. As far as I know, there are no mainstream bowling alleys that employ “pinsetters” for manual labor. Today, that term refers to the machines who do that work, while also keeping score automatically.

I can only imagine what it’s like to have your job fully automated, to the point where you get laid off. For some, that could be distressing, as it means having to find a new job, and maybe learn new skills. For others, it might end up being a relief, if they’re eventually able to move on to something more interesting or lucrative. In fact, many occupations have become obsolete throughout human history because of advancements in technology and social conditions.

However, not all automation technology seeks to eliminate jobs. Some technology enhances productivity (and possibly enjoyment!) of existing jobs. One thing that I’m particularly interested in is automating the boring parts of software engineering, so I can focus on the fun parts.

The fun parts of software engineering

If you’re anything like me, you got into computing because it’s fun and satisfying to solve problems with code. I like building things that are useful, and I like to share those solutions with others. I also especially like the idea of putting a computer to work for me, while I’m off doing other interesting things with my time.

This is actually how I got into Android app development specifically. There is something very… I dunno… cool about taking full control of a tiny computer, with internet access and sensors, on the go with me. Especially if I can make it do whatever I want, whenever I want, kind of like a Star Trek gadget. That’s fun to me (and I’m a bit of a geek because of it).

And the not-so-fun parts…

When it comes to professional software engineering, there is little tolerance for mistakes. Everyone makes mistakes, and that’s OK. But companies that charge money for people to use their software products tend to lose business if the software contains too many mistakes. Obviously, these companies put extra effort into making sure the mistakes don’t eventually affect the end users. That’s why we have software testing tools and frameworks.

While these tools are very helpful, from my experience, very few engineers who enjoy building software also truly enjoy testing it. Unless an organization requires engineers to write tests, the test code often gets skipped.

In one talk I used to give at conferences, I discussed mobile app quality tools offered by Firebase. I’d ask the audience: “Who here enjoys writing tests for your app? Raise your hand.” And the response is typically anywhere from zero to three people in a group of about 200.

Exhaustive testing is critically important, but it’s normally not all that fun. It’s usually a chore. Wouldn’t you love to have that task automated by a machine? Well, if you’re the one in the audience who raised their hand, maybe not! But for the rest of us, there’s an option.

Unit testing vs. system testing

Let me step back for a moment. There are various forms of testing, and types of code that an engineer writes to ensure code correctness. I really like pure unit testing as a way of validating that a unit of well-designed, encapsulated code does exactly what it’s expected to do. This type of testing is very effective to both ensure that the module under test is correct, and also for explaining the behavior of the module to other engineers. These tests run fast, and run frequently, in order to exercise and validate the code you’ve written, and give you immediate feedback.

Where things break down for me is with programmatic UI testing, which could be considered a form of system testing. The idea is to script the operation of the app, as if someone was using its UI, in order to ensure the different system components are working together. On Android, we typically use frameworks such as Espresso and UI Automator. For iOS, it’s XCTest. Or maybe you also use a framework such as Appium to cover multiple platforms.

Now, don’t get me wrong: I don’t think that system testing lacks value. I just don’t find it fun. There is nothing terribly interesting (to me) about writing code that digs into the UI of an app, finds buttons, checkboxes, menu items, etc, and manipulates them, then checks the UI again to see if the entire system did what it was supposed to do with that input. But I’ll say that it’s way better than repetitive manual testing!

The problems with writing UI tests

Aside from the lack of satisfaction I get from writing UI tests, here are some of the problems:

  • They can be kind of slow to run against a physical or virtual device, so they don’t necessarily give you instant feedback if something is broken.
  • They normally don’t tell you what lines of code were at fault, or even which system component is responsible, and it can even be hard to tell if the test itself was at fault. It’s still up to your team to dig in and root out the core issue.
  • System tests against a UI can also be kind of fragile, as they depend heavily on knowing intimate details about how the UI was constructed.

Take a look at this bit of Kotlin code from the Espresso home page that performs a very basic test with an EditText and a TextView widget, that should be wired up to deliver a message:

@Test fun greeterSaysHello() { onView(withId("Steve")) onView(withId( onView(withText("Hello Steve!")).check(matches(isDisplayed())) }

Here, you can see that you have to know the ID of the views to work with, such as “name_field” and “greet_button”. One small change to those IDs (even if working correctly), and the test will become broken immediately. (Even an IDE refactoring operation might not actually pick this up, which is a very real problem.)

Now, imagine if your job is to write system tests for some UI that you didn’t build. You might have to dig really far into the code to find all the widgets and IDs that you’ll need for the test. That’s just as tiring as fixing a nasty bug.

If your job is to write tests against a screen whose contents are unpredictable, it gets even worse. Let’s say the app’s backend generates random or ever-changing data to display in the app. If this backend can’t also be strictly controlled by the test, you could have a hard time figuring out how to validate that the UI is correct at any given moment, or write even more test code to get it right.

I won’t even get started about testing highly customized widgets, or making sure your displays are pixel perfect.

This is not fun at all! Wouldn’t you rather have a bot do all this for you? (I’m also talking to you, person who previously raised a hand! 🙋)

So, let that bot automate your system testing

I’d say that software engineers and testers would actually like to have a machine take over the less interesting parts of their jobs. This wouldn’t be like the pinsetters who had their jobs eliminated. Instead, a bot would augment the engineering roles. The engineers can then spend more time on the stuff that only engineers can do, such as find and implement optimizations, design better architecture, and find corner cases that break an app and its infrastructure.

If a bot could effectively learn how to use your app, and discover the use cases that are evident from its UI, it makes sense to let it go ahead and do exactly that.

That bot is Mesmer.

Using a combination of artificial intelligence and machine learning, Mesmer automatically crawls your app, recommends test cases, and runs them in batch to check for consistency. Since it uses computer vision to read the screen, it won’t easily be foiled by changing IDs. And it can cope with data that changes over time.

The number of lines of code to get started with Mesmer is exactly zero. All you need to do is upload your app, add some configuration, and let it go. Later, you can augment what it learns, adding your own assertions and scripts whenever you’re ready.

Mesmer aligns itself with a movement called Robotic Process Automation, or RPA. Similar to the pinsetting machines that improve efficiency and customer experience at a bowling alley, RPA involves AI-driven software bots improve the way your application development processes work, but without all the layoffs!

Does Mesmer’s automation mean I can skip writing UI tests entirely?

Absolutely not! There are some situations where writing UI tests is still critical. In particular, if you need to test a custom UI widget, it makes sense to isolate it on its own screen, and write code to interact with it, to make sure its internal changes in state are consistent with expectations. Or, if your app receives some sort of signal from the operating system, you’ll need a way to construct those situations, sometimes on actual devices.

Some real-world use cases are still really difficult to properly test, so we need frameworks like Espresso to drive the code that can’t be properly isolated and unit tested. And for that, we’ll still write UI tests.

But for everything else, we’ll let Mesmer’s bot do our jobs.



Mesmer is the leader in Application Experience Management. Our AI Bots evaluate the user experience of an app by acting like real users. Your development team uses us when they’re building and running their apps. We automate user experience testing & UAT testing as developers build their app. Then recommend accessibility improvements before they release it. And then monitor the experience once it’s released. The result- happier customers, faster releases, fewer problems.
Mesmer is headquartered in Palo Alto, California, and funded by Intel Capital and True Ventures.