How to choose the best experimentation solution

To build and deliver products that customers love, development teams at leading companies like Uber, Netflix, Airbnb, Google, and Facebook focus on experimentation as a critical business process. Engineers and product managers are empowered to A/B test each new product feature in a controlled environment to determine the impact to key metrics like engagement, product usage and revenue before launching the feature to everyone.

These companies have all built complex internal software platforms to support their experimentation-driven approach to product development. If you’re looking to adopt the same practice of validating digital product decisions with data, you’ll likely be evaluating how to implement an experimentation platform for your product and engineering team.

Depending on your company’s unique requirements and needs, there are a few paths you can take:

• Build your own server-side A/B testing framework in-house
• Integrate an open source solution
• Invest in a commercial experimentation platform

When considering your options, considering these four areas can help you get informed and determine which solution makes the most sense.

1. Statistical rigor: Do you have confidence in the system and the results it produces?

When implemented correctly, experimentation can help teams quickly identify which features and changes will drive business value, and which will fail. The best teams embrace failure, and try to fail as fast as possible so that they can move on to ideas that work. To "move fast and break things," in the words of Facebook’s founder, your teams need to trust that tests are being run correctly, and that the results are accurate.

However, many companies underestimate the difficulty of collecting data reliably, maintaining their analytics pipelines over time, and tracking results effectively. When events aren’t tracked correctly or analytics integrations stop working, it results in delays and slower experimentation velocity.

In many organizations, the responsibility for setting a sample size and analyzing results typically falls on an analytics or data science team. Some analytics teams who are responsible for analyzing every test find that they quickly become the bottleneck as experimentation scales across multiple teams.

When evaluating a platform for your company, ensure that your analyst and engineering teams are collaborating effectively and that your A/B testing framework is working well with your analytics stack. This will ensure that anyone can access accurate results in a timely fashion.

2. Ease of Use: Is the system easy for developers and accessible to business users?

The best teams in the world run thousands of experiments each year to maximize opportunities to learn. Usability for both technical and non-technical users can be the difference between running several experiments in a year and running hundreds or even thousands.

On most product teams, developers will be implementing tests directly within your codebase, so they often make the initial decision about which solution to start with. Many open source tools or home-grown systems operate entirely within code at the beginning.

This means that a deploy is required for each new change to the experiment, and that it’s difficult for product managers or analysts to understand what’s happening with running experiments without pulling data out of a database and analyzing reports by hand.

When determining whether a system will be accessible to business users, identify whether it contains remote configuration tools and easy-to-understand UI design. These features help teams increase experimentation velocity by decoupling experiments from deployment, and providing transparency into experiments across the organization.

An enterprise system should include the ability to start and stop an experiment, change traffic allocation, or update targeting conditions in real time from a central dashboard without a code deploy.

When it comes to developer productivity, focus on features of the solution like robust documentation and support for multiple languages that will enable the engineering team to spend less time figuring out how to run tests, and more time working on customer-facing feature work. A robust solution will also include APIs that can be used to automate certain task or integrate more deeply into development teams’ workflows.

3. Total cost of ownership: How will the system be developed and maintained over time?

Building in-house or adopting an open source framework typically comes with a relatively small upfront investment. Over time, however, additional features and customisations become necessary as more client-side and server-side teams use the platform, and maintenance burdens begin to distract engineers from core product focus.

To avoid unforeseen maintenance cost, ensure that in addition to understanding your team’s initial set of feature requirements, you’ve budgeted time for bug fixes and considered the more complex features you might need down the road.

Will you need to run experiments that are mutually exclusive, or use advanced targeting features? Determine how customizable these advanced features will be, and estimate the resources required for building and maintaining them.

Anticipating your organizational bandwidth for engineering and maintenance can help you plan beyond initial development, inform your decision to build or buy, and ensure critical features are available when your team needs them most.

You should also consider how various teams can experiment with various new features, without getting in each other’s way. A gatekeeping systems can help ensure that test groups don’t overlap.

4. People and process: Who will evangelize experimentation culture across your company?

Successful experimentation is as much about culture as it is about technology. The top teams invest in people and process to ensure that cultural change takes root, and that data-driven decision-making becomes the norm.

Whether you decide to build an experimentation platform in-house, or invest in a commercial solution, it’s critical to have the right people and processes in place.

The most successful teams in the world are those that have not just invested in internal tools and technologies, but have taken the time to build teams of data scientists, product managers and engineers, and created an internal culture of enablement that allows for experimentation-driven product development.

Identifying the members of your organization who will be responsible for defining key metrics and championing a culture of experimentation company-wide can help ensure you get the most out of whatever solution you choose.

Simon Herron, senior solutions engineer, Optimizely.

Published under license from ITProPortal.com, a Future plc Publication. All rights reserved.

Photo credit: ra2studio / Shutterstock