The Toolkit

A converts 120 of 1,000 visitors, B 135 of 1,000. With uniform priors, the posterior probability that B is better is 84.3%, and the expected loss from choosing B — how much conversion you give up on average in the worlds where A is better — is just 0.12 percentage points. A decision rule on expected loss can ship B even though a significance test would not reject.

• “Probability that B is better” is not immune to peeking: stopping when it crosses a line still raises error rates unless the decision rule accounts for it.
• Priors matter with little data; report how the conclusion changes under a sceptical prior.
• A posterior on conversion says nothing about long-term effects such as retention or novelty wearing off.

rng = np.random.default_rng(16)
a = rng.beta(1 + 120, 1 + 880, 200_000)              # 120 of 1,000 converted, uniform prior
b = rng.beta(1 + 135, 1 + 865, 200_000)              # 135 of 1,000
p_b_better = (b > a).mean()
expected_loss_b = np.maximum(a - b, 0).mean()        # what choosing B costs on average if wrong

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

# Python — Bayesian A/B test
from scipy.stats import beta
import numpy as np

# Observed data
a_conv, a_total = 50, 1000
b_conv, b_total = 60, 1000

# Posterior distributions (uniform prior)
post_a = beta(a_conv + 1, a_total - a_conv + 1)
post_b = beta(b_conv + 1, b_total - b_conv + 1)

# P(B > A) via simulation
samples = 100000
p_b_wins = (post_b.rvs(samples) > post_a.rvs(samples)).mean()
print(f"P(B > A) = {p_b_wins:.3f}")