My Experience with Snapshot Testing
Snapshot Testing — what is that?? I get questioned a lot about what it is and why we use it. It’s not as common as other types of testing such as Manual, Automated e2e, Regression, Integration, etc. In this article, i’ll discuss what snapshot testing is, when we have found it useful, when not so useful, some lessons learned, and existing tools.
What is Snapshot Testing?
Snapshot testing compares two images: a “Reference” image, which is the expected image of the application screen, to the “Current” image, which is the snapshot of what the screen is actually displaying. A “Difference” image is produced which displays where the two images are different.
Let’s take a look at what this looks like.
Here we have a simple screen with just a login button. The first screenshot is the reference image, the second screenshot is the current image, and the third screenshot is the difference image. You can easily see where your UI is no longer matching the expected UI design.
As you can see with these examples, it will pick up on color or text differences as well.
To Use or Not To Use Snapshot Testing
Snapshot Testing is very helpful in determining whether there are unexpected UI changes.
When determining whether or not to use snapshot testing, the main thing you should determine is how important UI changes matter to you… while remembering that although the changes may be minor, all those minor imperfections could make your application appear poorly. You should also consider if one code change could impact several areas of the application. If this is the case, UI changes could happen unintentionally often and may give you more reason to want to perform snapshot testing.
Depending on the automation e2e tool you are using, snapshot testing can be helpful to test items that you are otherwise unable to via the e2e tool. Here are some examples of when we used snapshot testing instead of iOS XCUITest e2e tests:
- Elements that do not show up in the hierarchy tree. In Xcode, you can inspect the current tree structure by running po XCUIApplication(). However, we have ran into issues where a few elements were not appearing like nav bar text, or error message text. In these instances we had no way to access the element via our XCUITests however, a snapshot test allowed us to verify these elements.
- To make sure the UI is correctly displayed still upon scrolling.
- Truncation. In the iOS hierarchy tree, the full text is always returned regardless of whether or not it is truncated on the UI.
- Colors. Obviously the snapshot tests will do color verification as shown in the example above. Some could argue why not do this color verification in the e2e tests instead. Some tools like Android Espresso allows you to do this; however you cannot using iOS XCUITest.
E2e automation testing can catch workflow, functionality, or text errors, but they aren’t going to tell you if it displayed in the wrong position on the screen. This is where snapshot testing comes into play. Other tools are meant to be used for performance or load testing — this is not the purpose of snapshot testing either.
Lessons Learned
- Snapshot Testing is not for everyone. However, if you want to ensure that you application maintains the expected design, use it. Otherwise, the quality of the design may degrade over time. Personally, when I notice an application that has a lot of design issues, I have less confidence in the functionality of the application. If they don’t take the time to care about the design, they also may not care much about the quality of the functionality.
- If you have a small, simple application where UI changes are very isolated to only affecting the intended target then snapshot testing wouldn’t be worth it.
- The reference images can take up a lot of space so ensure you are able to handle these and investigate if there are ways to generate smaller images that take up less space with the tool being used.
- Don’t over snapshot test. We initially started out with snapshot testing all of our supported iOS devices. We realized that when an issue occurred, it was typically either on all devices, or a device that was at one end on a screen size scale. We decided to cut back and only snapshot test the smallest and largest screen size devices. This allowed us to have a lot of confidence that almost all UI issues would still be found while greatly reducing our snapshot testing run time.
- Don’t snapshot frequently changing screens, data, or animations.
- When possible, only snapshot smaller elements or sub-sections of the screen. Do not include sections that are shared across several pages. For example, if you have an iOS navigation bar at the top or tab bar at the bottom, you would not want this to be included in every single one of your snapshots. Isolate the navigation bar and tab bar to their own snapshots, and only take a snapshot of the view inside or even just certain elements on that screen. This will alleviate you from needing to redo ALL snapshots when a change is made just to one of those areas.
Snapshot Testing Tools
iOSSnapshotTestCase
- https://github.com/uber/ios-snapshot-test-case
- This was originally developed by FaceBook and known as FBSnapshotTestcase but is now maintained by Uber.
- This is what we are currently using for our iOS Snapshot Testing. It has a lot of contributors so issues or updates are frequently addressed.
Snapshot Testing
- https://www.pointfree.co/blog/posts/23-snapshottesting-1-0-delightful-swift-snapshot-testing
- I recently stumbled upon this one. I haven’t tried it yet, but plan to look into this soon. I will post if I find it to be useful! It is written in Swift.
PixelTest
- https://github.com/KaneCheshire/PixelTest
- https://github.com/pointfreeco/swift-snapshot-testing
- Another one that I haven’t tried yet but also written in Swift.
Karumi Shot
- https://github.com/Karumi/Shot
- We used this for a little while for Android Snapshot Testing however we ran into issues often where it would think the screens were different but they weren’t and the difference images would only show a gray screen. It doesn’t support a tolerance value either like iOSSnapshotTestCase does. We ended up spending too much time investigating failures then ended up not really being failures. We stopped Snapshot Testing for Android at this time until this tool is more mature or we come across another tool.
Protractor-Image-Comparison
- https://github.com/wswebcreation/protractor-image-comparison
- We currently use this for snapshot testing of our web application via our protractor tests. However, as of July 2020, it will no longer be maintained.
If you have experience with any of the tools mentioned above or others, please comment and let me know your experience!
