Performance testing and a flexible managed service during transformational change

Phase 1: How performance engineering was used to overcome technical testing challenges

Phase 2: Managed service flexibility during transformational change

The client: A leading inter-dealer broker
Industry: Capital Markets
Technologies: Adobe Flex
Ten10 services: Performance Testing, Automated Testing, Agile Testing, Off-site Managed Service

Phase 1: Performance engineering and Technical Testing Troubleshooting

The Project

The client was developing a new trading platform, written in Adobe Flex. From the outset the importance of working with a specialist testing partner in order to get the performance and overall enduser experience right was well recognised.

Key Challenges

The new trading platform project involved a tremendous amount of change; different technology was being used to create the software on the customer’s desktop and mobile devices, the messaging technology changed from using RTMP (Adobe’s bespoke streaming data solution) to using Push Technology’s Diffusion product, and the backend serverside infrastructure was also changing.

Tackling change

Complications were identified at an early stage when it became clear that that the client’s internal performance testing tool of choice was not compatible with the new product. Following an internal tool review by the client, it had been clear that no other commercial tool would be able to test the product either.

Automation-at-scale

Ten10 was able to create a bespoke, development-led solution in order to meet the performance testing requirements. The solution allowed an “automation-at-scale” approach to be adopted, meaning the system could be load tested using cloud-based infrastructure. This approach provided the client with an additional benefit in that it also gave them important insights into the customer experience; allowing them to understand the true performance of their system and to identify when prices are actually displayed to the end user, depending on which geographical location they were based in across the world.

When performance problems were encountered in production - particularly the issue of users experiencing disconnects - the flexible approach of Ten10 meant that a technical team could be rapidly deployed onsite to investigate. The team worked logically to repeat and isolate the issue to determine the true cause.

Working together with the development teams within multiple suppliers and by examining supplier networking code and network infrastructure configuration the team was able to design creative and intelligent workarounds to solve the problem.

End-User Insights

During the testing process Ten10 was able to also identify a number of valuable insights for the client including;

• when failures occurred under load and how these are communicated to the end user;
• how the failures and recovery from the failures are observed real-time from the enduser perspective, rather than the traditional approach of just examining server-side failures retrospectively;
• identifying issues relating to scalability in the back-end system; and
• exposing a number of critical functional defects within the application which only materialise under load, both on the client application and server back-end systems.

Ten10 was able to present these findings back in a useful format by capturing application screenshots on failure, enabling the client to develop a much more in-depth understanding of the end-user experience.

These additional insights provided a complete picture of the system performance from an endto-end perspective, a true test of performance including how the application performed on the users’ desktop.

Phase 2: A fully flexible managed functional test service during transformational change

The project:

As the client looked to increase the size and depth of its flagship eCommerce trading platform, it required a scalable test team to provide ongoing functional testing.

Looking ahead the client was also preparing for a period of strategic transformational change, with a program underway to transform the Systems Development Lifecycle (SDLC) from a traditional waterfall/ITIL approach to an agile delivery model with continuous delivery.

Accelerating the move to Agile and DevOps

The automated testing introduced by Ten10 has been pivotal to the client’s success in making fast, incremental change as it has transitioned from a traditional waterfall/ITIL approach to an agile delivery model with continuous delivery.

Ten10’s considered approach to gaining a deep understanding of a client’s needs, activities and long-term product roadmap ensured that the choices made around the delivery of testing solutions met not only the requirements of each individual project, but also the needs of the business at a strategic IT level.

Ten10’s experience and specialist knowledge in agile testing has played a significant part in helping to accelerate the client’s move to Agile and DevOps as the client rolled out an ongoing program of change on a global scale for its key trading platforms.

Most recently Ten10 has demonstrated its value as a specialist testing partner by providing an Agile Testing training course, helping to further enhance the knowledge within the in-house test team as part of the overall transformation program.

This collaborative approach has enabled the client to benefit from the extensive experience and testing expertise that Ten10 brings; to successfully steer and guide them through the choices of tools, processes and techniques at both a strategic and on-the-ground level, whilst retaining resource flexibility that is so vital to ensure business objectives can be effectively met.

High quality testers with Capital Markets expertise

The client faced a number of challenges as they looked to hire an internal team based out of their central London office. The fast-moving nature of the business resulted in an ongoing need to adjust the size of the test team in very short-time frames. A permanent onsite team would restrict the flexibility needed to scaleup and scale-down the size of the test team in response to the high profile project’s changing requirements.

During previous recruitment drives the client had found it difficult to source high quality testers with the necessary blend of technology and capital markets expertise. People with this experience were in short supply and very expensive; and once hired, they often found that these testers were hard to retain given the competitive market within the City of London.

Another consideration for the client was the shortage of space in their office in the City of London. The associated high cost made it hard to justify building an internal test team when space could be better-allocated to revenue-generating trading desks.

Off-Site yet accessible

For over two years, Ten10 has provided a fully managed test service for its leading eCommerce trading platform.

Ten10 was able to meet the client’s demand for a multi-skilled team with experience of capital markets, providing seven consultants in total; one Managing Principal, one Lead Test Consultant, two Test Consultants and four Graduate Analysts. The initial test team were permanently based out of Ten10’s Leeds office, providing all the efficiencies of an off-site team, but with the important advantage of the team being in the same time zone.

The engagement was led by the Managing Principle with the day-to-day management of the team run by the Lead Test Consultant. This approach not only gave the client a consistent point of contact, but with the added benefit that as the consultants were located less than 3 hours away, they were able to attend the monthly Account Management meetings in person, held on site every month.

Following the success of this initial engagement the team expanded to 13 consultants working across four permanent projects and other business as usual work.