A/B Testing Methodology
Similar to any science-based methodology, you want to change one variable at a time and record the results. One variable being tested minimizes correlation and helps you define the variable being tested as causation. For instance, if you changed all your button colors on site, and you are measuring conversion, which button contributed the most to your success? Rather, if you changed only one button, you know that helped increase flow to the next page.
A/B Testing Regimin
Always keep a/b testing going. Have a schedule and a backlog that you can triage into upcoming tests. For a simple conversion funnel site with a defined goal, I preferred to have one test running at all times, and aimed to have a > 2.5% split for each test before declaring a winner or loser. My method for keeping my pipeline constantly going was in baseball terms:
- At Bat – Current test being run
- On Deck – The next test after the previous completes
- In the Hole – The test up after that one
- Backlog – The test ideas you have brainstormed, or better, what your enterprise has contributed to