Representation-Aware Experimentation: Group Inequality Analysis for A/B Testing and Alerting
arXiv (Cornell University)(2022)
Abstract
As companies adopt increasingly experimentation-driven cultures, it is crucial to develop methods for understanding any potential unintended consequences of those experiments. We might have specific questions about those consequences (did a change increase or decrease gender representation equality among content creators?); we might also wonder whether if we have not yet considered the right question (that is, we don't know what we don't know). Hence we address the problem of unintended consequences in experimentation from two perspectives: namely, pre-specified vs. data-driven selection, of dimensions of interest. For a specified dimension, we introduce a statistic to measure deviation from equal representation (DER statistic), give its asymptotic distribution, and evaluate finite-sample performance. We explain how to use this statistic to search across large-scale experimentation systems to alert us to any extreme unintended consequences on group representation. We complement this methodology by discussing a search for heterogeneous treatment effects along a set of dimensions with causal trees, modified slightly for practicalities in our ecosystem, and used here as a way to dive deeper into experiments flagged by the DER statistic alerts. We introduce a method for simulating data that closely mimics observed data at LinkedIn, and evaluate the performance of DER statistics in simulations. Last, we give a case study from LinkedIn, and show how these methodologies empowered us to discover surprising and important insights about group representation. Code for replication is available in an appendix.
MoreTranslated text
Key words
testing,group inequality analysis,alerting,representation-aware
PDF
View via Publisher
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Related Papers
Mitigating Urban-Rural Disparities in Contrastive Representation Learning with Satellite Imagery
2022
被引用1
Measures of Disparity and Their Efficient Estimation.
PROCEEDINGS OF THE 2023 AAAI/ACM CONFERENCE ON AI, ETHICS, AND SOCIETY, AIES 2023 2023
被引用0
Data Disclaimer
The page data are from open Internet sources, cooperative publishers and automatic analysis results through AI technology. We do not make any commitments and guarantees for the validity, accuracy, correctness, reliability, completeness and timeliness of the page data. If you have any questions, please contact us by email: report@aminer.cn
Chat Paper
GPU is busy, summary generation fails
Rerequest