Optimizely

Optimizely is a commercial experimentation and digital experience platform. It became well known for A/B testing on websites and has grown into a combined experimentation and feature management suite, used by marketing teams, product managers and data scientists to test and personalize what users see.

Its capabilities group into five. Experimentation: A/B tests, multivariate tests and multi-page experiments, with random assignment by cookie or ID, a visual editor for simple changes, and support for web, mobile and server-side tests. Feature management: feature flags that toggle behavior without redeploying, percentage rollouts, and targeting rules. Personalization: audience rules by location, device or behavior, and custom segments. Statistics: a frequentist engine with sequential-testing adjustments, reporting p-values, confidence intervals and effect sizes, with variance reduction available on higher tiers. Integration: connectors to analytics and CRM tools, and pipelines into warehouses and BI.

The workflow is the usual loop. State a hypothesis, configure the experiment, split traffic, track the events that matter, watch the results, end the test when they are conclusive under the sequential correction, and roll the winner out behind a feature flag.

Two of those steps deserve numbers rather than descriptions, because both are routinely read as something they are not.

What a rollout percentage can see, and what a dashboard of metrics costs

import numpy as np
from scipy.stats import norm

base, weekly = 0.05, 2_000_000
print(f"  {weekly:,} users a week, a {base:.0%} baseline, and a variant that is")
print(f"{'rollout':>10}{'users on it':>14}{'detectable at 80% power':>26}"
      f"{'power against -5%':>20}")
for pct in (0.01, 0.05, 0.10, 0.25, 0.50):
    n_t = int(weekly * pct)
    n_c = weekly - n_t
    se = np.sqrt(base * (1 - base) * (1 / n_t + 1 / n_c))
    mde = (norm.ppf(0.975) + norm.ppf(0.80)) * se
    power = norm.sf(norm.ppf(0.975) - 0.05 * base / se)
    print(f"{pct:>10.0%}{n_t:>14,}{mde / base:>25.2%}{power:>20.4f}")

rng = np.random.default_rng(0)
runs, n_met = 200_000, 20
print(f"\n  and a platform reports many metrics at once. {n_met} metrics, no real")
print(f"  effect in any of them, {runs:,} experiments:")
p = 2 * norm.sf(np.abs(norm.ppf(rng.random((runs, n_met)))))
any_raw = (p < 0.05).any(axis=1)
bonf = (p < 0.05 / n_met).any(axis=1)
srt = np.sort(p, axis=1)
thr = 0.05 * np.arange(1, n_met + 1) / n_met
bh = (srt <= thr).any(axis=1)
print(f"{'rule':>34}{'experiments with a finding':>29}")
print(f"{'any metric at 0.05':>34}{any_raw.mean():>29.4f}")
print(f"{'Bonferroni':>34}{bonf.mean():>29.4f}")
print(f"{'Benjamini-Hochberg at 0.05':>34}{bh.mean():>29.4f}")

z = norm.ppf(rng.random((runs, n_met)))
z[:, :5] += 3.0
p2 = 2 * norm.sf(np.abs(z))
srt2 = np.argsort(p2, axis=1)
ps = np.take_along_axis(p2, srt2, axis=1)
below = ps <= thr
kmax = np.where(below.any(axis=1), n_met - np.argmax(below[:, ::-1], axis=1), 0)
rank = np.arange(n_met)[None, :]
rej_bh = np.take_along_axis(rank < kmax[:, None], np.argsort(srt2, axis=1), axis=1)
print(f"{'rule':>34}{'true effects found':>21}{'false ones reported':>22}")
for name, rej in (("no correction", p2 < 0.05),
                  ("Bonferroni", p2 < 0.05 / n_met),
                  ("Benjamini-Hochberg", rej_bh)):
    print(f"{name:>34}{rej[:, :5].sum(axis=1).mean():>21.4f}"
          f"{rej[:, 5:].sum(axis=1).mean():>22.4f}")
#   2,000,000 users a week, a 5% baseline, and a variant that is
#    rollout   users on it   detectable at 80% power   power against -5%
#         1%        20,000                    8.68%              0.3647
#         5%       100,000                    3.96%              0.9424
#        10%       200,000                    2.88%              0.9982
#        25%       500,000                    1.99%              1.0000
#        50%     1,000,000                    1.73%              1.0000
#
#   and a platform reports many metrics at once. 20 metrics, no real
#   effect in any of them, 200,000 experiments:
#                               rule   experiments with a finding
#                 any metric at 0.05                       0.6412
#                         Bonferroni                       0.0483
#         Benjamini-Hochberg at 0.05                       0.0495
#                               rule   true effects found   false ones reported
#                      no correction               4.2535                0.7507
#                         Bonferroni               2.4526                0.0371
#                 Benjamini-Hochberg               3.2602                0.1738

Two readings

The first block separates feature management from experimentation, which the single product interface tends to blur. In this traffic, baseline and effect setting, a 1% rollout is a containment device: it limits how many users meet a bad variant, and it has 0.3647 power against a 5% regression, so it will miss that regression about two weeks in three. That is not a flaw — containment is what it was for — but a non-significant result at low power gives weak reassurance against this regression. Larger total traffic or effects can make even a small rollout informative; 5% is not a universal testing threshold.

The second and third blocks are about the dashboard. A platform reports many metrics per experiment, and under a complete null, watching twenty of them at 5% produces a finding 64% of the time. Both standard corrections hold that near 5%; they differ once something is real. Bonferroni controls the probability of one or more false rejections in the specified hypothesis family and loses real findings to do it — 2.45 of five true effects found against 4.25 uncorrected. Benjamini–Hochberg controls the expected false discovery proportion under its assumptions, not the realized fraction in every experiment, and recovers 3.26 of the five. Neither is the right answer in general. Which one a platform applies, and whether it applies one at all, changes what a green cell on the dashboard means, and it is worth knowing before reading one.

The sequential correction the workflow mentions belongs to the same family of decisions. The published method behind always-valid inference for continuously monitored A/B tests is Johari, Pekelis and Walsh’s, which derives p-values that stay valid under continuous monitoring and extends the idea to multiple hypothesis testing in the sequential setting. When an experiment runs on a sequential statistics engine, that is what makes “monitor significance in real time” a safe instruction rather than the peeking problem it would otherwise be. In a fixed-horizon analysis the same instruction is the peeking problem.

Where it fits

The advantages are practical. Non-technical teams can run experiments without engineering time. Client-side and server-side testing are both supported. Feature flags come with the product rather than as a second system. The statistical engine handles sequential monitoring rather than leaving it to the reader. And it connects to the analytics and marketing tools already in place.

The limitations are equally practical. It is commercial SaaS and priced accordingly against open-source alternatives. It depends entirely on metric definitions being right, which is work the tool cannot do. And for genuinely unusual experiments — novel randomization units, complex interference structures — an internal platform is more flexible.

Typical uses follow the capability list: pricing, checkout flows and product pages in e-commerce; content layout in media; gradual feature rollout and UI changes in SaaS; landing pages and creatives in marketing. Against neighbors in the same space, VWO is aimed at marketing teams but lighter for engineering use, Adobe Target is enterprise-grade with deep personalization and a heavier setup tied to the Adobe stack, and LaunchDarkly is strong on feature flagging for developers; it analyzes the metric events it hosts itself and can also use metrics computed in an external data warehouse.

One caveat applies to every sentence above that names a feature. Vendor products change, and tiers change faster than product names; the capability list here reflects the state described when this entry was written, and current documentation is the authority on what a given plan includes today.

References

• Ramesh Johari, Leo Pekelis and David J. Walsh, “Always Valid Inference: Bringing Sequential Analysis to A/B Testing”, arXiv:1512.04922. arxiv.org
• SciPy documentation, scipy.stats.norm. docs.scipy.org