What is it about?
Digital experiments are routinely used to test the value of a treatment relative to a status quo control setting. The treatment effect could be different across major sub-groups based on user characteristics. We propose a framework for detecting and analyzing these heterogeneities in treatment effects in digital experiments. Analysis of 27 real-world experiments spanning 1.76 billion sessions and simulated data demonstrates the effectiveness of our framework relative to existing techniques.
Featured Image
Photo by Markus Spiske on Unsplash
Why is it important?
As digital experiments have become increasingly pervasive in organizations and a wide variety of research areas, their growth has prompted new challenges for experimentation platforms. One challenge is that experiments often focus on the average treatment effect (ATE) without explicitly considering differences across major sub-groups — heterogeneous treatment effect (HTE). This is especially problematic because ATEs have decreased in many organizations as the more obvious benefits have already been realized.
Perspectives
We show the need to discover heterogeneities in treatment effects based on the data from a large digital experimentation platform. I hope this encourages the development of more features on experimentation platforms to explore heterogeneity so that various stakeholders, like product managers, analysts, and data scientists, can engage in data-driven decision-making.
Sriram Somanchi
University of Notre Dame
As lead author Sriram Somanchi eloquently noted, in academic research and practice (e.g., industry data science and product teams), there is a need to move beyond mostly relying on average treatment effects (ATE). Digital experiments are about learning and data-driven decision-making. Analysis of heterogenous treatment effects (HTEs), using frameworks such as the one we propose, can provide deeper insights about how major sub-groups react to a proposed digital artifact/intervention. These interventions include new search relevance, recommender system, engagement, advertising, and other digital user experience (DUX) machine learning (ML) innovations. Often times, the value of these innovations varies for different sub-groups. HTE aligns nicely with the shift towards developing and testing novel personalized ML/experiences.
Ahmed Abbasi
University of Notre Dame
Read the Original
This page is a summary of: Examining User Heterogeneity in Digital Experiments, ACM Transactions on Information Systems, January 2023, ACM (Association for Computing Machinery),
DOI: 10.1145/3578931.
You can read the full text:
Contributors
The following have contributed to this page
