Crowdsourced mobile testing of html email templates

Crowdsourced mobile testing of html email templates

The client: An international law firm with over 2500 lawyers in offices around the world
Industry: Legal
Technologies: CRM
Ten10 services: Test strategy, mobile testing, functional testing

The client

An international legal firm with high ethical standards and a passion for helping its clients succeed, the client uses creativity to find innovative solutions to difficult challenges and explain them simply. It also adds value through its pro bono work, its volunteering and its ethical and responsible business practices.

Summary

The client was developing email templates for marketing purposes. The aesthetics of these templates were very important to ensure that they looked attractive and would be rendered correctly on a multitude of potential recipient devices.

Ten10 enabled the client to quickly test its templates against a very wide range of email client hardware and software by finding the right individual for the project, creating the methodology, and organising and managing the entire process.

The project

The client uses a large CRM system, and relies on a number of email templates for their marketing and communications. As a result the client had invested heavily in the aesthetics of the templates but needed to ensure they were rendering correctly and displaying attractively as intended. Therefore the templates needed to be tested against a wide range of hardware as well as email client software.

The law firm’s normal approach to testing would be to identify a defined set of applications for testing. However, with email marketing templates the variables include; device type, email client, operating systems and browsers. With so many considerations the project moved beyond the scope of the Operation Team’s resources.

Goals

Ten10 was asked to step in and help to find a solution quickly. Rather than harvest a considerable amount of different devices, which would have been very time-consuming, Ten10 chose to deliver a crowdsourced testing service. The methodology entailed entering a list of email clients that had been identified for testing. We then sent emails to those addresses and asked the recipients to return screenshots and notes on the email’s appearance to provide information on the look and feel of the email. Had there been a requirement for repeat testing then it would have been worth setting up a dedicated lab; however for a one-time test this was not required.

We wanted to minimise the client’s workload. All the client needed to do was to provide the emails, together with the possibility of iterating the design in the event of the template not appearing satisfactorily.

Challenges

We needed to be able to demonstrate sufficient coverage of testing in a very short timescale, which meant quickly finding individuals with domain knowledge.

The biggest challenge, however, was that there were no metrics available from the CRM tool to understand which email clients and browsers were being used to read the marketing material. This lack of data meant that significant client coverage was required for compatibility testing. A normal test lab would not contain the variety of devices required to provide the client with enough confidence that the email templates would be rendered correctly for all recipients, so Ten10 needed to find a more innovative solution.

Success story

Ten10 has the largest database of qualified, UK-based testing contractors. Drawing on this comprehensive database of Ten10 Associates meant that we could quickly and confidently present the client with candidates for the task, who would be suitable both for the engagement, and for the client. The client made us aware that we had understood their needs far more quickly than other agencies that they had previously worked with, meaning that we not only achieved a high success rate at providing the right candidate but managed to do so at pace.

Once the human resource requirement was in place, we provided the methodology to organise and manage the entire process. Understanding that this was a one-time test, we took the pragmatic and innovative approach of using our distributed test resource to provide the coverage and speed the client was looking for.

Our cloud-based, crowdsourced testing solution enabled us to identify sufficient devices to provide coverage and quickly distribute the testing effort to the people involved. Emails were sent, covering the various templates. These were captured as a screenshot by the testers, and then uploaded to a central location for the client to validate.

Ten10 satisfied the client’s requirement with a very quick solution to their challenge of ensuring that marketing emails could be distributed with the design looking as intended, irrespective of the hardware and software combination being used by their recipients.

The legacy resulting from this project is enhanced experience and resource in this area, and a strong relationship with a satisfied client.