Probability, distributions, hypothesis testing, Bayesian inference and A/B testing for ML engineers.
Key Concepts to Know
Practice Statistics with AI
Timed session with instant scoring, voice support, and model answers.
1 Interview Questions
Browse all topics →When is a p-value less than 0.05 still not enough to ship a change?
Model Answer
P < 0.05 means "low probability of seeing this effect if the null is true" — it does NOT mean the effect is large, durable, or worth shipping. Reasons to NOT ship despite significance: (1) Effect size is tiny — statistical significance with no business value. (2) Multiple comparisons inflated false-positive rate — apply Bonferroni or FDR correction. (3) Novelty effect — users react to "new" but interest fades; rerun after 2 weeks. (4) Confounders — A/B was during a holiday, on a specific segment, etc. (5) Pre-registration violated — you tested 10 metrics and one was significant. (6) Cost — does the engineering cost to maintain the change justify the lift?
Related Topics