Setting Up Snapshot Testing with Paparazzi in a Modularized, Multi-Flavor Android Project using Azure DevOps

Disclaimer: This article isn't about how Paparazzi works. If you're new to Paparazzi, I suggest checking out their official repository.

Very briefly, for those unfamiliar with Paparazzi, it's an open-source snapshot testing library for Android. Snapshot testing involves capturing a "snapshot" of your UI components and comparing it against a reference image to detect unintended changes. Paparazzi works by rendering your Android views on the JVM, without needing an emulator or physical device. It generates an image of the rendered view, which can then be compared to a previously approved reference image. This approach allows for fast, reliable UI testing that can catch unexpected visual changes early in the development process.

Not so long ago, I had to set up a snapshot testing pipeline in our modularized, multi-flavor Android project. By "multi-flavor," I mean we have one codebase released in 3 different countries as completely different apps. By "modularized," I mean each feature set within an application lives under its own feature module.

Context: Each module is owned and maintained by a team, and there is no dedicated platform team to handle all the CI/CD pipelines. Therefore, each team needs to have some knowledge about CI/CD to be able to deliver features independently and efficiently as much as possible.

Without further ado, let's get started. We have a typical yaml file under the azure folder of the project which contains all the necessary configuration steps for the pipeline to run on Azure DevOps. Steps for each flavor are run in parallel independently.

Given the situation, we decided to have a separate yaml file (snapshot-test-pipeline.yml) under each module to give each team autonomy in managing their snapshot tests end-to-end for their own modules. This approach allows teams to customize their testing process as needed while maintaining consistency across the project. Here's how it looks:

parameters:
  country: ''

steps:
  - task: Gradle@2
    displayName: "Snapshot paparazzi test run"
    inputs:
      tasks: "{moduleName}:verifyPaparazzi${{ parameters.country }}DevDebug"
      publishJUnitResults: true
      testResultsFiles: '**/TEST-*.xml'

  - task: PublishBuildArtifacts@1
    displayName: "Publish snapshot delta images"
    condition: or(eq(variables['Agent.JobStatus'], 'Failed'), eq(variables['Agent.JobStatus'], 'SucceededWithIssues'))
    inputs:
      PathtoPublish: "$(System.DefaultWorkingDirectory)/moduleName/build/paparazzi/failures"
      ArtifactName: "SnapshotDeltaImages"

As mentioned above, in the main pipeline.yml file under the azure folder, we invoke snapshot-test-pipeline.yml as a template for each module separately. For instance, pipeline.yml would look like this:

# Make sure to invoke templates at the right place. For example, right after running your unit tests
...
 - template: /path/to/module1/snapshot-test-pipeline.yml
   parameters:
      country: 'France'

- template: /path/to/module2/snapshot-test-pipeline.yml
  parameters:
      country: 'France'
...

...
 - template: /path/to/module1/snapshot-test-pipeline.yml
   parameters:
      country: 'Germany'

- template: /path/to/module2/snapshot-test-pipeline.yml
  parameters:
      country: 'Germany'
...

This setup allows us to run snapshot tests for each module and country combination independently, providing flexibility and maintainability for our multi-flavor, modularized project.

That's it! This approach has helped us manage snapshot testing effectively across our complex project structure, allowing each team to maintain their tests while ensuring consistency in the overall pipeline.

Please share your thoughts, and if you have a similar setup but are approaching it differently, let me know in the comments. I would love to learn from your experiences!