R1: You need to diagnose an error in the program:
The google maps team wants to understand whether dismiss rate is a reasonable metric to help understand user experience of a button in the app. The hypothesis is that, the higher the dismiss rate, the worse the user experience. Hence, they perform a simulation in the A/A comparison scenario. In the simulation, signal = all interactions on the button (click, dismiss, ignore, ...), and negative signal = dismiss.
The pseudo code is as follows. Note that we might refer to the numerator and denominator often in later discussions.
result_pval = []
for replica in (1:1000):
# the number of overall signals follows a roughly bell shaped distribution
num_signal_control = round(random.normal(150, std = 30))
num_signal_treatment = round(random.normal(150, std = 30))
# given the number of overall signals, the number of negative signals follows a binomial distribution
num_negative_signal_control = random.binomial(num_signal_control, 0.5)
num_negative_signal_treatment = random.binomial(num_signal_treatment, 0.5)
# define numerator and denominator of the test statistics
# the idea of the denominator is: we use Normal approximation to estimate the variance of the numerator
p_hat_control = num_negative_signal_control / num_signal_control
p_hat_treatment = num_negative_signal_treatment / num_signal_treatment
numerator = p_hat_treatment - p_hat_control
denominator = sqrt(
p_hat_treatment*(1-p_hat_treatment)/num_signal_treatment
+ p_hat_control *(1-p_hat_control) /num_signal_control
)
testing_statistics = numerator / denominator
# calculate p value and append to the result vector
p_value = 2*std_normal_area_under_curve(
lower = abs(testing_statistics), upper = infinity)
result_pval = append(result_pval, p_value)
plot_histogram(result_pval)
The histogram of the p-values is skewed to the right on [0,1]. In other words, there are more p values < 0.5 than p values > 0.5.
Q1: Is such a distribution of p-value expected?
