Validating UX Design Hypotheses through A/B Tests

As UX researchers, we often encounter a common challenge: deciding whether one design truly outperforms another. Maybe one version of an interface feels faster or looks cleaner. But how do we know if those differences are meaningful - or just the result of chance? To answer that, we turn to statistical comparisons. When comparing numeric metrics like task time or SUS scores, one of the first decisions is whether you’re working with the same users across both designs or two separate groups. If it's the same users, a paired t-test helps isolate the design effect by removing between-subject variability. For independent groups, a two-sample t-test is appropriate, though it requires more participants to detect small effects due to added variability. Binary outcomes like task success or conversion are another common case. If different users are tested on each version, a two-proportion z-test is suitable. But when the same users attempt tasks under both designs, McNemar’s test allows you to evaluate whether the observed success rates differ in a meaningful way. Task time data in UX is often skewed, which violates assumptions of normality. A good workaround is to log-transform the data before calculating confidence intervals, and then back-transform the results to interpret them on the original scale. It gives you a more reliable estimate of the typical time range without being overly influenced by outliers. Statistical significance is only part of the story. Once you establish that a difference is real, the next question is: how big is the difference? For continuous metrics, Cohen’s d is the most common effect size measure, helping you interpret results beyond p-values. For binary data, metrics like risk difference, risk ratio, and odds ratio offer insight into how much more likely users are to succeed or convert with one design over another. Before interpreting any test results, it’s also important to check a few assumptions: are your groups independent, are the data roughly normal (or corrected for skew), and are variances reasonably equal across groups? Fortunately, most statistical tests are fairly robust, especially when sample sizes are balanced. If you're working in R, I’ve included code in the carousel. This walkthrough follows the frequentist approach to comparing designs. I’ll also be sharing a follow-up soon on how to tackle the same questions using Bayesian methods.

Recently, someone shared results from a UX test they were proud of. A new onboarding flow had reduced task time, based on a very small handful of users per variant. The result wasn’t statistically significant, but they were already drafting rollout plans and asked what I thought of their “victory.” I wasn’t sure whether to critique the method or send flowers for the funeral of statistical rigor. Here’s the issue. With such a small sample, the numbers are swimming in noise. A couple of fast users, one slow device, someone who clicked through by accident... any of these can distort the outcome. Sampling variability means each group tells a slightly different story. That’s normal. But basing decisions on a single, underpowered test skips an important step: asking whether the effect is strong enough to trust. This is where statistical significance comes in. It helps you judge whether a difference is likely to reflect something real or whether it could have happened by chance. But even before that, there’s a more basic question to ask: does the difference matter? This is the role of Minimum Detectable Effect, or MDE. MDE is the smallest change you would consider meaningful, something worth acting on. It draws the line between what is interesting and what is useful. If a design change reduces task time by half a second but has no impact on satisfaction or behavior, then it does not meet that bar. If it noticeably improves user experience or moves key metrics, it might. Defining your MDE before running the test ensures that your study is built to detect changes that actually matter. MDE also helps you plan your sample size. Small effects require more data. If you skip this step, you risk running a study that cannot answer the question you care about, no matter how clean the execution looks. If you are running UX tests, begin with clarity. Define what kind of difference would justify action. Set your MDE. Plan your sample size accordingly. When the test is done, report the effect size, the uncertainty, and whether the result is both statistically and practically meaningful. And if it is not, accept that. Call it a maybe, not a win. Then refine your approach and try again with sharper focus.

How to Approach A/B Testing as a Data Analyst A/B testing is a great way to help make data driven decisions on whatever project or product you may be working on.  Here’s a step by step setup guide for how you can go about creating and analyzing A/B tests. This example is mainly focused on doing an A/B test in an ecomm site, but the general principles apply regardless. 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝘆𝗼𝘂𝗿 𝗚𝗢𝗔𝗟 𝗮𝗻𝗱 𝗮 𝗗𝗼𝗰𝘂𝗺𝗲𝗻𝘁: Before doing any tech work, you need to clearly understand what you’re trying to accomplish from the test. Make a document outlining the test and set a clear objective in a doc that exactly states what the goal of the A/B test is - are you trying to increase CVR from testing a new feature, encourage repeat rates, etc. What ever the objective is - make a doc outlining the test and start at the top with clearly writing down the goal, then write down your whole testing plan. 𝗠𝗮𝗸𝗲 𝘆𝗼𝘂𝗿 𝗛𝘆𝗽𝗼𝘁𝗵𝗲𝘀𝗶𝘀: In the same doc you state the GOAL - right after it, write down what your test hypothesis is. This really just is, what change do you expect or think you will see fro your test. Here’s an example: Changing the color of the add-to-cart button from green to red, will increase ATC rate by 10%. 𝗦𝗲𝗴𝗺𝗲𝗻𝘁 𝗬𝗼𝘂𝗿 𝗔𝘂𝗱𝗶𝗲𝗻𝗰𝗲: Divide your test population into smaller groups, for an A/B usually 50,50 but if your testing 2 variables could be 33/33/33%. Each sub group you make assign in the Testing doc, which variation of the test will the group get, either control or variant. 𝗗𝗼 𝘁𝗵𝗲 𝘁𝗲𝗰𝗵 𝘄𝗼𝗿𝗸 𝘁𝗼 𝗰𝗿𝗲𝗮𝘁𝗲 𝘁𝗵𝗲 𝘃𝗮𝗿𝗶𝗮𝗻𝘁𝘀: Now you actually have to hookup in the backend to direct your site traffic to receive either the control group or test group that you’ve defined in the Testing doc. Usually you’re going to work with a frontend engineer to make sure all the code is hooked up and ready to go. 𝗥𝘂𝗻 𝘁𝗵𝗲 𝗧𝗲𝘀𝘁: Kick off the test. Make sure you let the test run long enough for statistical significance to be reached. 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝗞𝗲𝘆 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: Before kicking off the test, at least make sure you have all you need to collect the data to measure the results on the test. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝘁𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀: Do a through analysis of all the data that answers the question. Did the change in the variant group lead to a statistically significant improvement over the control? Make sure to validate with stat tests. 𝗥𝗲𝗰𝗼𝗺𝗺𝗲𝗻𝗱 𝗮 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻: Make a recommendation and document it in your testing doc, using data as evidence to support if you should implement the change in your Variant group or stay using the tech in the control group. And in a nutshell, that’s how you do an A/B test, this is just a high level overview of it. Overall patience in data collection and precision in the GOAL of the test are key for a successful A/B test.