A/B testing & AI recommendations

A/B testing

Experiments test popup variants against each other to find what converts.

• Deterministic bucketing. A visitor is assigned to a variant by hashing visitor_id + experiment_id (SHA-256), so the same visitor always sees the same variant and the split is stable.
• Significance, not vibes. Winners are declared with a two-proportion z-test, and each variant's conversion rate carries a Wilson 95% confidence interval, so you ship the winner when the data supports it, not when the dashboard looks promising on a Friday.

Run experiments on headline, offer, form length, or trigger, and let the math arbitrate.

AI recommendations

On a periodic cadence, an LLM analysis compares your funnel, attribution, and popup performance against industry benchmarks and returns ranked recommendations, flagged critical / high / medium / info, so you always have a prioritised next move.

• The analysis is prompt-injection guarded.
• In multi-process deployments, a distributed lease elects a single leader so the analysis runs once, not once per process.

Workflow

1. Form a hypothesis from a funnel drop or a replay.
2. Run an A/B experiment on the relevant popup.
3. Ship the variant the z-test declares.
4. Let AI recommendations surface the next opportunity.