A Protocol to Reduce Bias & Variance in Head-to-Head Tests

A Protocol to Reduce Bias & Variance in Head-to-Head Tests

Head-to-head tests are an effective way to compare the performance of competing approaches. Practical constraints imposed by platforms such as Google AdWords prevent us from using a simple user based split in running these head-to-head tests. Hence, we have developed a new protocol: a geographical split of the total population into two sub-populations allocated to each competitor and time-based swaps of these populations between the two competitors. The proposed protocol is more robust to variability in sales due to seasonal, regional or other latent effects as opposed to simpler splits based only on geography and/or without time-based swaps.
We provide theoretical proof and show empirically that both features are necessary to reduce bias and variance in the estimation of performance of advertising campaigns. We also prove on a simplified model that increasing the number of splits and swaps reduces the bias and variance.