Strategic A/B testing via Maximum Probability-driven Two-armed Bandit

TY - JOUR

T1 - Strategic A/B testing via Maximum Probability-driven Two-armed Bandit

AU - Zhang, Yu

AU - Zhao, Shanshan

AU - Wan, Bokui

AU - Wang, Jinjuan

AU - Yan, Xiaodong

N1 - Publisher Copyright: © 2025 by the author(s).

PY - 2025

Y1 - 2025

N2 - Detecting a minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have a significant economic impact. Traditional methods, reliant on normal distribution-based or expanded statistics, often fail to identify such minor effects because of their inability to handle small discrepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes a maximum probability-driven twoarmed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation methods further enhances the robustness and efficacy. The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hypothesis and a less concentrated one under the alternative hypothesis, greatly improving statistical power. The experimental results indicate a significant improvement in the A/B testing, highlighting the potential to reduce experimental costs while maintaining high statistical power.

AB - Detecting a minor average treatment effect is a major challenge in large-scale applications, where even minimal improvements can have a significant economic impact. Traditional methods, reliant on normal distribution-based or expanded statistics, often fail to identify such minor effects because of their inability to handle small discrepancies with sufficient sensitivity. This work leverages a counterfactual outcome framework and proposes a maximum probability-driven twoarmed bandit (TAB) process by weighting the mean volatility statistic, which controls Type I error. The implementation of permutation methods further enhances the robustness and efficacy. The established strategic central limit theorem (SCLT) demonstrates that our approach yields a more concentrated distribution under the null hypothesis and a less concentrated one under the alternative hypothesis, greatly improving statistical power. The experimental results indicate a significant improvement in the A/B testing, highlighting the potential to reduce experimental costs while maintaining high statistical power.

UR - https://www.scopus.com/pages/publications/105023497734

M3 - Conference article

AN - SCOPUS:105023497734

SN - 2640-3498

VL - 267

SP - 77069

EP - 77089

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

T2 - 42nd International Conference on Machine Learning, ICML 2025

Y2 - 13 July 2025 through 19 July 2025

ER -