A/B Testing within an Organization

A/B Testing for Big Business

I got my start with A/B testing within a large organization with a simple, 1-product funnel. The business I worked for lived and died by a single subscription product and each acquisition was worth a LOT of money annually. Because of the way this business was set up, conversion funnel optimizations were directly attributable to annualized revenue performance. A single product funnel allows one to really avoid the noise caused by product variability, price ranges, merchandising, etc. Every change on the site was measured via testing before it was committed. Not only does increased performance result in increased revenue, a poorly performing test results in saving the company a revenue loss.

A/B Testing Basics

The basics of A/B Testing and optimization of your conversion funnel is using javascript to change elements on page for each user.

In a basic 50/50 A/B test, 50% of your users will see one experience and 50% will see the other experience. Most tools, at the very basic level, will identify which user is which via a cookie, then serve them a several snippets of javascript and html (including CSS) that manipulate the experience on page. It then passes the cookie as a flag on the user and records behavior based on what you’ve defined as your goals. As users flow through these separate experiences, metrics are generated and statistical confidence increases with volume. Defining what ‘confidence’ means at the beginning of your test is encouraged, rather than making stories to rationalize the results.

Leading A/B Testing

One of the things I’ve picked up in various companies is that A/B testing is only successful when championed and made exciting to leadership. Packaging testing Wins and Losses as revenue gains and revenue saves is exciting. Landing page testing for ad campaigns is exciting. I urge people to schedule regular results meetings with various tests that includes learnings. These methods become more exciting when other businesses adopt the methodologies you use in their world. Get the enterprise involved in creating test ideas and a way for people to easily submit tests into the backlog. Keeping results posted also helps keep your colleagues visually interested.