A/B Testing

Existing Solutions

Despite the challenges of A/B testing, there are multiple solutions that have emerged to help businesses conduct effective experiments.

A/B Testing Options

Existing solutions for A/B testing run the gamut. There are bare-bones apps exposing APIs for creating and retrieving experiments from the command line. In addition, there are many What You See Is What You Get (WYSIWYG) options for performing A/B testing on client-side-rendered applications that can be used with limited developer expertise (e.g., VWO). At the other end of the spectrum are full featured A/B testing solutions that offer exposed APIs as well as SDKs for both server-side and client-side rendered applications that require significant developer expertise.

Options for A/B testing solutions that are open source with self-hosting options are relatively few – applications that look promising either have little to no development or engagement over time or they become fully-featured enterprise options that offer only expensive, managed solutions (e.g., LaunchDarkly, Optimizely).

Selected A/B Testing Providers and Features

Self-Hosted OptionNoneNoneDockerDocker, NodeNone
Managed OptionYesYesYesYesRequired
Analytics Data StorageNoNoNoNoYes
Client Data Store RequiredNoNoYesNoNo
Developer Expertise RequiredYesNoYesYesYes
Client- or Server-SideClientClientBothBothBoth
GUI for Test CreationYesYesYesYesPro or Enterprise Plan
GUI for Data VisualizationYesYesYesNoPro or Enterprise Plan

Within the open-source, self-hosted category, two key options are Unleash and GrowthBook, both of which offer rich documentation and API access, with open-source deployment options. These solutions inspired some of the key features and functionality of Test Lab.

Test Lab Comparison

Challenges of A/B Testing